GDS - General Data Structure
A General Data Structure (GDS) is a universal, composable data structure and is used to store any kind of data.
Typical usage is the definition of configurations, specifications and data sets.
- Introduction
- A Hash and an Array
- The GDS Language
- An Example Using the GDS Language
- Motivation and Features
- Another Example
- Advanced Example
- Usage with Ruby
- Syntax Definition
- Variables
- String Interpolation
- Schema Specifiers
- References
- Classic Ruby Syntax
- Language Extensions
- Limitations
- Example: Conversion to JSON
- Example: Conversion to XML
- Comparing to YAML
- Practical Examples
- Implementation
- Ruby Gem
- Source Code
Introduction
In this special definition a General Data Structure (GDS) is either a hash or an array or any composition of nested hashes and arrays. A GDS is used to store any kind of structured data.
The defined data can be used in the following situations:
- for the configuration or specification of systems, processes, services, frameworks, objects, methods or functions
- for seeding a database or any other data model
- to convert to other data representation formats like JSON, XML, YAML
A Hash and an Array
In Ruby you have to use curly braces to define hashes and to group nested hashes. In Ruby you also have to use square brackets to define arrays and to group nested arrays.
A Hash:
A hash is a collection of unique keys and their values - it is a collection of key-value pairs.
h = { key1: 'value a', key2: 'value b' }
An Array:
An array is an ordered collection of any object.
a = [ 1, 2, 3, 4 ]
The GDS Language
The GDS language is a special DSL (domain specific language) for defining general data structures.
It uses a succinct, indentation-sensitive syntax which makes data representation clear and readable.
The building blocks for general data structures are hashes and arrays.
An Example Using the GDS Language
,
: name Karl
addresses ,
: street Ringstr. 5 | zipcode 77777 | city Saarbrücken
: street Zwirbelweg 8 | zipcode 61112 | city Stuttgart
transforms to
[
{ name: 'Karl',
addresses: [ { street: 'Ringstr. 5' , zipcode: 77777, city: 'Saarbrücken' },
{ street: 'Zwirbelweg 8', zipcode: 61112, city: 'Stuttgart' } ]
}
]
Motivation and Features
- the focus is on the essential data
- a language for humans, a human-writable format
- structure and hierarchy is expressed using indentation (whitespace-sensitive) - avoids curly braces and square brackets
- uses a succinct and minimalist syntax - tries to avoid unnecessary characters - minimizes visual noise
- in many cases colons, commas and quotes are not necessary
- less text makes data clearer and more readable
- for configuration and specification - replacement for XML, JSON, YAML
- for data/database seeding
- less text and less potential errors using schema definitions
- supports block comments, which could be nested
- allows also the definition of hash and array structures in a kind of restricted classic Ruby like syntax
- provides an alternative for Ruby hash and array definition without using eval(); can be used as a protection against code injection vulnerabilities, e.g. on web servers
Another Example
The next example demonstrates the definition of a nested hash.
To simplify the definition of a top-level hash structure, you can omit the first : (colon) for the definition of the main hash.
This makes your definition more pleasant to read.
caption foo
credit bar
images
small
url http://mywebsite.com/image-small.jpg
dimensions
height 500
width 500
large
url http://mywebsite.com/image-large.jpg
dimensions
height 500
width 500
videos
small
preview http://mywebsite.com/video.m4v
dimensions
height 300
width 400
transforms to
{
:caption => 'foo',
:credit => 'bar',
:images => {
:small => {
:url => 'http://mywebsite.com/image-small.jpg',
:dimensions => {
:height => 500,
:width => 500
}
},
:large => {
:url => 'http://mywebsite.com/image-large.jpg',
:dimensions => {
:height => 500,
:width => 500
}
},
},
:videos => {
:small => {
:preview => 'http://mywebsite.com/video.m4v',
:dimensions => {
:height => 300,
:width => 400
}
}
}
}
Advanced Example
This example is using a schema specifier.
@schema person(firstname,lastname,age)
, @schema person
: Harry | Langemann | 44
: Susi | Heimstett | 32
: Bob | Meiermann | 57
transforms to
[
{ firstname: 'Harry', lastname: 'Langemann', age: 44 },
{ firstname: 'Susi', lastname: 'Heimstett', age: 32 },
{ firstname: 'Bob', lastname: 'Meiermann', age: 57 },
]
Alternative definition with GDS without using schema specifiers:
,
:
firstname Harry
lastname Langemann
age 44
:
firstname Susi
lastname Heimstett
age 32
:
firstname Bob
lastname Meiermann
age 57
Another alternative definition with GDS:
,
: firstname Harry | lastname Langemann | age 44
: firstname Susi | lastname Heimstett | age 32
: firstname Bob | lastname Meiermann | age 57
Usage with Ruby
Installation
gem install gdstruct
Coding
h = { a: 'val a', b: 'val b' }
can be coded as
require 'gdstruct'
h = GDstruct.c( <<-EOS )
a val a
b val b
EOS
and
a = [ 1, 2, 3, 4 ]
can be coded as
require 'gdstruct'
a = GDstruct.c( <<-EOS )
,
1
2
3
4
EOS
Use the class GDstruct and the class method c (alias for create). This method expects a string with the description. In the example the string is defined as a here document.
Syntax Definition
The GDS language uses two basic symbols (: and ,) for the creation of hashes and arrays.
| : | A colon is used to define a hash. |
| , | A comma is used to define an array. |
Use indentation with two spaces for the definition of elements and for nested structures. Tab characters are not allowed for indentation.
Definition of a hash
| : | Use a colon to define a hash. |
Usually each element of a hash needs to be defined on a separate line and needs to be indented one level higher (two spaces more) than the defining ':' (colon) symbol.
The elements of a hash are key-value pairs.
The value part of a key-value pair could be a subhash (a nested hash), an array or a basic value.
:
# => {}
:
k1 v1
k2 v2
k3 v3
# => { k1: 'v1', k2: 'v2', k3: 'v3' }
By default, the key is always converted into a symbol.
You also can already start the definition of key-value pairs at the line of the hash definition with the colon.
The following definition is equivalent to the previous one:
: k1 v1
k2 v2
k3 v3
# => { k1: 'v1', k2: 'v2', k3: 'v3' }
Attention: Between the hash defining : (colon) and the following key (in this case k1) there needs to be at least one space character!
This is always the case if the : (colon) for defining a hash and the following first key of the hash are on the same line.
Nested hashes:
You can define a nested hash, if you specify on a line only a key and no value.
The key-value pairs of this nested hash have to be defined on new lines and have to be indented one level more (two spaces more).
If you don’t define elements for a nested hash, then this hash stays empty.
:
k1
# => { k1: {} }
:
k1 v1
k2 v2
k3
k31 v31
k32
k33
k331 v331
k332
k3321
k4 v4
# => { k1: 'v1', k2: 'v2', k3: { k31: 'v31', k32: {}, k33: { k331: 'v331', k332: { k3321: {} } } }, k4: 'v4' }
You also can start the definition of a nested hash at the same line as the definition of the containing hash, that means at the same line as the hash definition with the colon (‘:’). The following is valid syntax:
: k1
# => { k1: {} }
Also this is valid syntax:
: k1
k11 v11
# => { k1: { k11: "v11" } }
The default structure is a hash
If the overall structure of the definition is a hash, then you can omit the first : (colon) for the definition of the main hash.
This means instead of writing
:
k1 v1
k2 v2
k3
k31 v31
k4 v4
# => { k1: 'v1', k2: 'v2', k3: { k31: 'v31' }, k4: 'v4' }
You can write it just like this:
k1 v1
k2 v2
k3
k31 v31
k4 v4
Please note that the default structure of GDS is a hash.
Multiple key-value pairs on a single line
| | | Use a vertical bar symbol to separate multiple key-value pairs on a single line. |
There is a more compact notation for the definition of key-value pairs supported.
You can define multiple key-value pairs on one single line.
The vertical bar symbol (|) is used to separate the individual key-value pairs.
This notation is only possible if the value part of the key-value pair is a basic value like a string, an integer, …
:
k1 v1 | k2 v2 | k3 v3
# => { k1: 'v1', k2: 'v2', k3: 'v3' }
The following definition is equivalent to the previous one:
: k1 v1 | k2 v2 | k3 v3
# => { k1: 'v1', k2: 'v2', k3: 'v3' }
One more example:
k1 v1 | k2 v2
k3 v3
k4 v4
k5 v5 | k6 v6
k7 v7 | k8 v8 | k9 v9
# => { k1: 'v1', k2: 'v2', k3: 'v3', k4: 'v4', k5: 'v5', k6: 'v6', k7: 'v7', k8: 'v8', k9: 'v9' }
Definition of keys
The standard syntax for the key is like for an identifier in most programming languages.
The first character needs to be a letter or an underscore ([A-Za-z_]) and can be followed by a sequence of
letters, underscores and digits ([A-Za-z0-9_]).
If you would like to use any other characters inside a key, then you need to quote the key with single quotes or double quotes.
As already mentioned, the key is always converted into a symbol.
Please take a look at the following examples:
k1 v1
'k2' v2
"k3" v3
" " empty
":" :colon
'$' dollar sign
' a really long key ' " very special "
'& %$$! §' something cryptic
# => { :k1=>"v1", :k2=>"v2", :k3=>"v3", :" "=>"empty", :":"=>:colon, :"$"=>"dollar sign", :" a really long key "=>" very special ", :"& %$$! §"=>"something cryptic" }
Definition of an array
| , | Use a comma to define an array. |
You can nest an array structure as deep as you like. The following definitions are all valid:
,
# => []
,
,
# => [ [] ]
,
,
,
# => [ [ [] ] ]
,
,
,
,
# => [ [ [ [] ] ] ]
Usually each element of an array needs to be defined on a separate line and needs to be indented one level higher (two spaces more) than the defining ',' (comma) symbol.
,
v1
v2
,
v31
,
,
v32
,
v331
v332
# => [ 'v1', 'v2', [ 'v31', [], [], 'v32', [ 'v331', 'v332' ] ] ]
Multiple values on a single line
| | | Use a vertical bar symbol to separate multiple values on a single line. |
There is a more compact notation for the definition of values supported.
You can define multiple values of an array on one single line.
The vertical bar symbol (|) is used to separate the individual values.
This notation is only possible if the values are basic value like a string, an integer, …
,
v1 | v2 | v3
# => [ 'v1', 'v2', 'v3' ]
,
v1 | v2 | v3
v4 | v5
v6
# => [ 'v1', 'v2', 'v3', 'v4', 'v5', 'v6' ]
, v1 | v2 | v3
v4 | v5
v6
# => [ 'v1', 'v2', 'v3', 'v4', 'v5', 'v6' ]
Definition of a hash containing an array
| <key> , | define an array within a hash, the array is the value for the <key>. |
k1 v1
k2 ,
v21
v22
v23
k3 v3
# => { k1: 'v1', k2: [ 'v21', 'v22', 'v23' ], k3: 'v3' }
Definition of an array containing a hash
Array containing hashes with single key-value pair
| : <key> <value> | define a hash with a single key-value pair |
,
: k1 v1
: k2 v2
# => [ { k1: 'v1' }, { k2: 'v2' } ]
Array containing hashes with multiple key-value pairs
| : <newline> | define a hash |
| <indentation +2> <key> <value> | with multiple key-value pairs |
| … |
,
:
k11 v11
k12 v12
:
k21 v21
k22 v22
# => [ { k11: 'v11', k12: 'v12' }, { k21: 'v21', k22: 'v22' } ]
If you don’t increment the indentation after the colons, then the following would be recognized:
,
:
k11 v11
k12 v12
:
k21 v21
k22 v22
# => [ {}, 'k11 v11', 'k12 v12', {}, 'k21 v21', 'k22 v22' ]
General example
,
v1
: k2 v2
: k31 v31 | k32 v32 | k33 v33
k34 v34
k35 v35 | k36 v36
:
k41 v41
k42 v42
k43,
k44, 1 | 2 | 3
4 | 5
6
7
8 | 9
: k441 v441
k5 :
k51 v51
k52 v52
:
k6
k61 v61
v7
# => [ 'v1', { k2: 'v2' }, { k31: 'v31', k32: 'v32', k33: 'v33', k34: 'v34', k35: 'v35', k36: 'v36' },
{ k41: 'v41', k42: 'v42', k43: [], k44: [1, 2, 3, 4, 5, 6, 7, 8, 9, { k441: 'v441' } ] },
{ k5: { k51: 'v51', k52: 'v52' } }, { k6: { k61: 'v61' } }, 'v7' ]
Basic Values
Basic values are integer numbers, floating-point numbers, symbols (Ruby symbols), strings and special keyword values like nil, true and false.
Basic values are used inside the definition of hashes and arrays.
In hashes the value part of a key-value pair could be a subhash (nested hash), an array or a basic value.
In arrays the values could be subarrays (nested arrays), hashes or basic values.
The following literals are available as notations for basic values:
Integer Literals
Integer values are defined like the Ruby syntax.
Take a look at the following examples.
Decimal representation:
- 1
- -1
- +1
- 0d1
- -0d1
- +0d1
Hexadecimal representation:
- 0xff
- 0x1001
- -0xBB
Binary representation:
- 0b0
- 0b1
- 0b00
- 0b11
- 0b0_0
- 0b0_0_0
- 0b1_0
- 0b1_0_0
Octal representation:
- 0o0
- 00
- 0o27
- 027
Floating-Point Literals
Floating-point values are defined like the Ruby syntax.
Examples
- 1.0
- -1.0
- +1.0
- 1.0e2
- 1.0E2
- 1.0e-3
- 1_000.000_000_1
In contrast to Ruby the following syntax is also valid:
- .1
- -.2
- +.3
Keyword Literals
Keyword literals are specific for the underlying programming language, in this case Ruby.
In the GDS language keyword literals are prefixed with an at sign (@), like directives.
The following keyword literals are supported.
| Literal | Representation in Ruby |
|---|---|
| @true | true |
| @false | false |
| @nil | nil |
Ruby Symbols
You can also use Ruby symbols as a basic value.
The literal has to be prefixed with a colon (:) which is followed by a sequence of arbitrary characters except any space characters.
As an example take a look at the following list of symbols:
,
:a
:longerstring
:longer_string
:@
:$
:$;
:$,
:$,,
:π
:'
:''
:"
:""
::
:::
::::
# => [ :a, :longerstring, :longer_string, :"@", :"$", :$;, :$,, :"$,,", :π, :"'", :"''", :"\"", :"\"\"", :":", :"::", :":::" ]
You can use symbols also inside a hash as the value part of a key-value pair.
k
k1 :asymbol
k2 :$;
k3 :@
k4 :$,,
k5 ::::
# => { k: { k1: :asymbol, :k2=>:$;, k3: :"@", k4: :"$,,", k5: :":::" } }
String Literals
You can use single quotes or double quotes to express string literals.
Examples
- ‘this is a string’
- “this is a string”
Escaping
Single-quoted strings use the following escaping
| \’ | ’ |
Double-quoted strings use the following escaping
| \” | ” |
The following escaping is used for both, single- and double-quoted strings
| \n | line feed character, newline character |
Default String Literals
In case the value is not recognized as an integer, a floating-point, a keyword, a symbol or a quoted string, then as a default it will be a string.
The definition of the string lasts until the end of the line or until a vertical bar character (|) appears or until a comment starts. The string will not include any leading or trailing space characters. However the string will contain inner space characters.
,
mystring
this is a really long string
first string : (1) | second string : (2) /*comment*/ | third string : (3) # comment
# => [ "mystring", "this is a really long string", "first string : (1)", "second string : (2)", "third string : (3)" ]
Comments
The GDS language supports inline comments and block comments.
Inline Comments
Inline comments use the Ruby style inline comment with an beginning # character.
Inline comments can be used after some valid input, e.g. after the definition of a value, at the end of a line. If inline comments are used on its own on a line, without any other valid input, then they can be indented with any number of spaces. The GDS syntax is not enforcing proper indentation for lines containing only inline comments. This is in contrast to some other indentation-sensitive languages, like for example HAML, and makes the usage of comments more handy.
,
v1 # inline comment at the end of a line
# inline comment with 0 space indentation
# inline comment with 1 space indentation
# inline comment with 2 space indentation
# inline comment with 3 space indentation
# inline comment with 4 space indentation
# ...
v2
# => [ 'v1', 'v2' ]
Block Comments
Block comments use the C style block comment syntax with opening /* characters and closing */ characters.
Block comments offer greater flexibility compared to Ruby’s multi-line comments (=begin, =end).
Block comments in GDS can be nested.
You can use block comments before, after or around values or key-value pairs. But please pay attention with block comments before a value (array) or a key-value pair (hash): You have to respect the proper indentation level! Otherwise either the value or key-value pair would be nested wrongly or a syntax error would occure.
If block comments are used on its own on a line, without any other valid input, then they can be indented with any number of spaces. The GDS syntax is not enforcing proper indentation for lines containing only block comments.
:
k1 v1
/* bc (block comment), attention: respect proper indentation of this line */ k2 /*bc*/ v2 /*bc*/
/* block comment with 0 space indentation */
/* block comment with 1 space indentation */
/* block comment with 2 space indentation */
/* block comment with 3 space indentation */
/* block comment with 4 space indentation */
/* ... */
/****
multi-line block comment
****/
/*bc (respect indentation)*/ k3 /*bc*/ , /*bc*/
/*bc (respect indentation)*/ 1 /*bc*/
/*bc (respect indentation) /*nested bc*/ */ /*bc*/ 2 /*bc*/ | /*bc*/ 3 /*bc*/
/*bc (respect indentation)*/ k4 /*bc*/ , /*bc*/ 1 /*bc*/ | /*bc*/ 2 /*bc*/
# => { k1: 'v1', k2: 'v2', k3: [1,2,3], k4: [1,2] }
The following definitions all contain wrong indentations for lines with a beginning block comment and a definition of a value after that. All of these definitions would result in a syntax error. Effectively it is the same as using a wrong indentation for the definition of a value without a beginning block comment.
,
v1
/*bc !!! wrong indentation !!! */ v2
,
v1
/*bc !!! wrong indentation !!! */ v2
,
v1
/*bc !!! wrong indentation !!! */ v2
Variables
You can define a variable as a placeholder for a basic value: for an integer, a floating-point, a keyword literal, a symbol or a string.
A variable name is an identifier (unique name) and has to be prefixed with $. Variables are defined at the beginning of a definition of a GDS construct, before any hash or array structures are defined.
$integer = 10
$float = 20.3
$symbol = :sym$%&
$string = this is a default string
$string_sq = 'single-quoted string'
$string_dq = "double-quoted string"
$flag = @true
$something = @nil
k1 $integer
k2 $float
k3 $symbol
k4 $string
k5 $string_sq
k6 $string_dq
k7 $flag
k8 $something
valuelist ,
$integer
$float
$symbol
$string
$string_sq
$string_dq
$flag
$something
# => { k1: 10, k2: 20.3, k3: :"sym$%&", k4: "this is a default string", k5: "single-quoted string",
# k6: "double-quoted string", k7: true, k8: nil,
# valuelist: [ 10, 20.3, :"sym$%&", "this is a default string", "single-quoted string", "double-quoted string", true, nil ] }
If a variable is used which was not defined before, then an exception is raised.
String Interpolation
In conjunction with variables, there is string interpolation supported.
The values of variables are substituted into string literals.
$object = house
setup "This is a $(object)."
# => { setup: "This is a house." }
String interpolation takes place for double-quoted strings and for default strings, however not for single-quoted strings.
You can prevent string interpolation taking place by escaping the $( sequence with a backslash \.
$errorCode = 301
$errorText = Moved Permanently
messages
response1 "The error was: $(errorText); The error code is $(errorCode)."
response2 The error was: $(errorText); The error code is $(errorCode).
response3 'The error was: $(errorText); The error code is $(errorCode).' # no string interpolation in single-quoted string
response4 The error was: \$(errorText); The error code is $(errorCode). # prevent string interpolation by escaping with backslash \
# => { messages: {
# response1: "The error was: Moved Permanently; The error code is 301.",
# response2: "The error was: Moved Permanently; The error code is 301.",
# response3: "The error was: $(errorText); The error code is $(errorCode).",
# response4: "The error was: $(errorText); The error code is 301." } }
During string interpolation the values of variables are first converted to strings and then they are substituted.
$integer = 10
$float = 20.3
$symbol = :$hello$
$true = @true
$false = @false
$nil = @nil
values ,
The integer number is $(integer)
The float number is $(float)
The symbol is $(symbol)
True is $(true)
False is $(false)
And nil really is nothing: $(nil)
# => { values: [
# "The integer number is 10",
# "The float number is 20.3",
# "The symbol is $hello$",
# "True is true",
# "False is false",
# "And nil really is nothing: " ] }
One more example
$main_path = /usr/john
$service_dir = docker-compose-service01
$app01_dir = application_001
$app02_dir = application_002
$app03_dir = application_003
$service_path = $(main_path)/$(service_dir)
config
app01_path $(main_path)/$(service_dir)/$(app01_dir)
app02_path $(main_path)/$(service_dir)/$(app02_dir)
app03_path $(service_path)/$(app03_dir)
# => { config: {
# app01_path: "/usr/john/docker-compose-service01/application_001",
# app02_path: "/usr/john/docker-compose-service01/application_002",
# app03_path: "/usr/john/docker-compose-service01/application_003" } }
Schema Specifiers
Schema specifiers can be used to predefine the keys of a hash or subhash.
When you specify the values you no longer need to name the key for each value.
This facilitates the input for hashes and prevents typos for keys.
In combination with the vertical bar symbol (|), which allows to define multiple values on a single line,
this features a slim and clearly arranged, table-like style for data.
@schema country(name,capital,area,population,vehicleRegistrationCode,iso3166code,callingCode)
, @schema country /*
----------------------------------------------------------------------------------------------------------------------------
name capital area (km^2) population vehicleRegistrationCode iso3166code callingCode
---------------------------------------------------------------------------------------------------------------------------- */
: Deutschland | Berlin | 357_385 | 82_521_653 | D | DE | 49
: USA | Washington, D.C. | 9_833_520 | 325_719_178 | USA | US | 1
: China | Beijing | 9_596_961 | 1_403_500_365 | CHN | CN | 86
: India | New Dehli | 3_287_469 | 1_339_180_000 | IND | IN | 91
: Austria | Vienna | 83_878 | 8_822_267 | A | AT | 43
: Denmark | Copenhagen | 42_921 | 5_748_769 | DK | DK | 45
: Canada | Ottawa | 9_984_670 | 36_503_097 | CDN | CA | 1
: France | Paris | 643_801 | 66_991_000 | F | FR | 33
: Russia | Moscow | 17_075_400 | 144_526_636 | RUS | RU | 7
transforms to
[
{ :name=>"Deutschland", :capital=>"Berlin", :area=>357385, :population=>82521653, :vehicleRegistrationCode=>"D", :iso3166code=>"DE", :callingCode=>49 },
{ :name=>"USA", :capital=>"Washington, D.C.", :area=>9833520, :population=>325719178, :vehicleRegistrationCode=>"USA", :iso3166code=>"US", :callingCode=>1 },
{ :name=>"China", :capital=>"Beijing", :area=>9596961, :population=>1403500365, :vehicleRegistrationCode=>"CHN", :iso3166code=>"CN", :callingCode=>86 },
{ :name=>"India", :capital=>"New Dehli", :area=>3287469, :population=>1339180000, :vehicleRegistrationCode=>"IND", :iso3166code=>"IN", :callingCode=>91 },
{ :name=>"Austria", :capital=>"Vienna", :area=>83878, :population=>8822267, :vehicleRegistrationCode=>"A", :iso3166code=>"AT", :callingCode=>43 },
{ :name=>"Denmark", :capital=>"Copenhagen", :area=>42921, :population=>5748769, :vehicleRegistrationCode=>"DK", :iso3166code=>"DK", :callingCode=>45 },
{ :name=>"Canada", :capital=>"Ottawa", :area=>9984670, :population=>36503097, :vehicleRegistrationCode=>"CDN", :iso3166code=>"CA", :callingCode=>1 },
{ :name=>"France", :capital=>"Paris", :area=>643801, :population=>66991000, :vehicleRegistrationCode=>"F", :iso3166code=>"FR", :callingCode=>33 },
{ :name=>"Russia", :capital=>"Moscow", :area=>17075400, :population=>144526636, :vehicleRegistrationCode=>"RUS", :iso3166code=>"RU", :callingCode=>7 }
]
which could be pretty written in classic Ruby syntax like
[
{ name: "Deutschland", capital: "Berlin" , area: 357_385, population: 82_521_653, vehicleRegistrationCode: "D" , iso3166code: "DE", callingCode: 49 },
{ name: "USA" , capital: "Washington, D.C.", area: 9_833_520, population: 325_719_178, vehicleRegistrationCode: "USA", iso3166code: "US", callingCode: 1 },
{ name: "China" , capital: "Beijing" , area: 9_596_961, population: 1_403_500_365, vehicleRegistrationCode: "CHN", iso3166code: "CN", callingCode: 86 },
{ name: "India" , capital: "New Dehli" , area: 3_287_469, population: 1_339_180_000, vehicleRegistrationCode: "IND", iso3166code: "IN", callingCode: 91 },
{ name: "Austria" , capital: "Vienna" , area: 83_878, population: 8_822_267, vehicleRegistrationCode: "A" , iso3166code: "AT", callingCode: 43 },
{ name: "Denmark" , capital: "Copenhagen" , area: 42_921, population: 5_748_769, vehicleRegistrationCode: "DK" , iso3166code: "DK", callingCode: 45 },
{ name: "Canada" , capital: "Ottawa" , area: 9_984_670, population: 36_503_097, vehicleRegistrationCode: "CDN", iso3166code: "CA", callingCode: 1 },
{ name: "France" , capital: "Paris" , area: 643_801, population: 66_991_000, vehicleRegistrationCode: "F" , iso3166code: "FR", callingCode: 33 },
{ name: "Russia" , capital: "Moscow" , area: 17_075_400, population: 144_526_636, vehicleRegistrationCode: "RUS", iso3166code: "RU", callingCode: 7 }
]
which could be pretty written in GDS style with no schema specifier like
,
: name Deutschland | capital Berlin | area 357_385 | population 82_521_653 | vehicleRegistrationCode D | iso3166code DE | callingCode 49
: name USA | capital Washington, D.C. | area 9_833_520 | population 325_719_178 | vehicleRegistrationCode USA | iso3166code US | callingCode 1
: name China | capital Beijing | area 9_596_961 | population 1_403_500_365 | vehicleRegistrationCode CHN | iso3166code CN | callingCode 86
: name India | capital New Dehli | area 3_287_469 | population 1_339_180_000 | vehicleRegistrationCode IND | iso3166code IN | callingCode 91
: name Austria | capital Vienna | area 83_878 | population 8_822_267 | vehicleRegistrationCode A | iso3166code AT | callingCode 43
: name Denmark | capital Copenhagen | area 42_921 | population 5_748_769 | vehicleRegistrationCode DK | iso3166code DK | callingCode 45
: name Canada | capital Ottawa | area 9_984_670 | population 36_503_097 | vehicleRegistrationCode CDN | iso3166code CA | callingCode 1
: name France | capital Paris | area 643_801 | population 66_991_000 | vehicleRegistrationCode F | iso3166code FR | callingCode 33
: name Russia | capital Moscow | area 17_075_400 | population 144_526_636 | vehicleRegistrationCode RUS | iso3166code RU | callingCode 7
As you see, the lines are getting very long and there is a lot of text to read. Clarity declines.
Schema specifiers are defined at the beginning of a definition of a GDS construct, before any hash or array structures are defined. You use the directive @schema, an identifier (unique name) for the schema and, within parentheses, a comma-separated list of key names.
@schema schema1(key1,key2,key3)
Schemas definitions are used for the construction of an array of hashes. You use a schema definition by using the @schema directive again and naming the schema specifier. This has to be located after the comma symbol (,) for the definition of an array. This can happen in two cases:
- for the definition of a top-level array or a nested array
# top-level array
, @schema schema1
: val1_key1 | val1_key2 | val1_key3
: val2_key1 | val2_key2 | val2_key3
...
# nested array
,
, @schema schema1
: val1_key1 | val1_key2 | val1_key3
: val2_key1 | val2_key2 | val2_key3
...
- for the definition of an array which is related to a hash key inside the definition of a hash
:
key , @schema schema1
: val1_key1 | val1_key2 | val1_key3
: val2_key1 | val2_key2 | val2_key3
...
As the schema defines the layout or structure of a hash, each line defining the values has to start with the colon symbol (:) for the definition of a hash.
If a schema specifier is used which was not defined before, then an exception is raised.
Extending schema definitions with additional data
Each hash created with a schema definition can be extended with additional information, that means with additional key-value pairs.
This keeps data definition flexible.
Please note, the definition of additional key-value pairs needs to start in a new line.
@schema person(firstname,lastname,age)
, @schema person
: Harry | Langemann | 44
hobby bicycling # additional data
favoriteFood Spaghetti | favoriteColor blue # additional data
: Susi | Heimstett | 32
: Bob | Meiermann | 57
transforms to
[
{ firstname: 'Harry', lastname: 'Langemann', age: 44, hobby: 'bicycling', favoriteFood: 'Spaghetti', favoriteColor: 'blue' },
{ firstname: 'Susi', lastname: 'Heimstett', age: 32 },
{ firstname: 'Bob', lastname: 'Meiermann', age: 57 }
]
Nested schema definitions
An example with two schema specifiers, where one is used nested to the other:
@schema p(firstname, lastname, phone1, phone2, email, web) # person
@schema a(street, zipcode, city, country) # address
, @schema p
: May | Grimes | '1-916-595-1175' | '(690) 557-4123' | may@connelly.name | https://www.ryanwolff.net
addresses , @schema a
: Grayce Mall 4833 | 77071 | East Rebeka | Cocos (Keeling) Islands
: Abshire Turnpike 297 | 94595 | Dasiaberg | Cocos (Keeling) Islands
: Ortiz Shores 29441 | 56813 | Windlerbury | Anguilla
: Legros Village 621 | 42101 | North Flavio | Cape Verde
: Jammie | Beahan | '(717) 922-5565' | '(345) 161-3052' | jammie@dicki.biz | https://www.corkerylind.org
addresses , @schema a
: Rosalyn Ports 2019 | 35316 | East Willachester | Democratic People's Republic of Korea
: Rosario Locks 56923 | 11183 | North Marionbury | Mozambique
: Mariane | Roob | '(749) 095-8597' | '(783) 183-1409' | mariane@fay.com | https://www.bosco.co
addresses , @schema a
: Dickens Pine 89210 | 24325 | Lake Alessandraton | Nicaragua
: Kamren Corners 742 | 98862 | Brionnamouth | Guadeloupe
: Erik Branch 67028 | 56464 | Kertzmannstad | Kyrgyz Republic
: Ansel | Braun | '(162) 247-8471' | '303-108-0535' | ansel@heidenreich.name | https://www.quitzondickens.net
addresses , @schema a
: Littel Path 650 | 26250 | Lorenzoton | Saint Helena
In Ruby syntax it would be like this:
[
{ firstname: "May", lastname: "Grimes", phone1: "1-916-595-1175", phone2: "(690) 557-4123", email: "may@connelly.name", website: "https://www.ryanwolff.net",
addresses: [
{ street: "Grayce Mall 4833" , zipcode: 77071, city: "East Rebeka" , country: "Cocos (Keeling) Islands" },
{ street: "Abshire Turnpike 297", zipcode: 94595, city: "Dasiaberg" , country: "Cocos (Keeling) Islands" },
{ street: "Ortiz Shores 29441" , zipcode: 56813, city: "Windlerbury" , country: "Anguilla" },
{ street: "Legros Village 621" , zipcode: 42101, city: "North Flavio" , country: "Cape Verde" }
], },
{ firstname: "Jammie", lastname: "Beahan", phone1: "(717) 922-5565", phone2: "(345) 161-3052", email: "jammie@dicki.biz", website: "https://www.corkerylind.org",
addresses: [
{ street: "Rosalyn Ports 2019" , zipcode: 35316, city: "East Willachester" , country: "Democratic People's Republic of Korea" },
{ street: "Rosario Locks 56923" , zipcode: 11183, city: "North Marionbury" , country: "Mozambique" }
], },
{ firstname: "Mariane", lastname: "Roob", phone1: "(749) 095-8597", phone2: "(783) 183-1409", email: "mariane@fay.com", website: "https://www.bosco.co",
addresses: [
{ street: "Dickens Pine 89210" , zipcode: 24325, city: "Lake Alessandraton", country: "Nicaragua" },
{ street: "Kamren Corners 742" , zipcode: 98862, city: "Brionnamouth" , country: "Guadeloupe" },
{ street: "Erik Branch 67028" , zipcode: 56464, city: "Kertzmannstad" , country: "Kyrgyz Republic" }
], },
{ firstname: "Ansel", lastname: "Braun", phone1: "(162) 247-8471", phone2: "303-108-0535", email: "ansel@heidenreich.name", website: "https://www.quitzondickens.net",
addresses: [
{ street: "Littel Path 650" , zipcode: 26250, city: "Lorenzoton" , country: "Saint Helena" }
], },
]
You have to decide which style to prefer. If you want to explicitly write key-value pairs or if you want to use schema specifiers.
I agree, the last example with nested schema specifiers could be somewhat dizzying.
However, it gets clearer if you use a different layout:
@schema p(firstname, lastname, phone1, phone2, email, website) # person
@schema a(street, zipcode, city, country) # address
, @schema p /*
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
firstname lastname phone1 phone2 email website street zipcode city country
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- */
: May | Grimes | '1-916-595-1175' | '(690) 557-4123' | may@connelly.name | https://www.ryanwolff.net
addresses , @schema a
: Grayce Mall 4833 | 77071 | East Rebeka | Cocos (Keeling) Islands
: Abshire Turnpike 297 | 94595 | Dasiaberg | Cocos (Keeling) Islands
: Ortiz Shores 29441 | 56813 | Windlerbury | Anguilla
: Legros Village 621 | 42101 | North Flavio | Cape Verde
: Jammie | Beahan | '(717) 922-5565' | '(345) 161-3052' | jammie@dicki.biz | https://www.corkerylind.org
addresses , @schema a
: Rosalyn Ports 2019 | 35316 | East Willachester | Democratic People's Republic of Korea
: Rosario Locks 56923 | 11183 | North Marionbury | Mozambique
: Mariane | Roob | '(749) 095-8597' | '(783) 183-1409' | mariane@fay.com | https://www.bosco.co
addresses , @schema a
: Dickens Pine 89210 | 24325 | Lake Alessandraton | Nicaragua
: Kamren Corners 742 | 98862 | Brionnamouth | Guadeloupe
: Erik Branch 67028 | 56464 | Kertzmannstad | Kyrgyz Republic
: Ansel | Braun | '(162) 247-8471' | '303-108-0535' | ansel@heidenreich.name | https://www.quitzondickens.net
addresses , @schema a
: Littel Path 650 | 26250 | Lorenzoton | Saint Helena
Data is not available - @na directive
In the case the value for a corresponding key is not available, you can use the @na directive (not available) to express this circumstance.
The resulting hash does not contain a key-value pair for this key.
@schema person( firstname, lastname, age )
, @schema person
: Harry | Langemann | 44
: @na | Heimstett | 32
: Bob | Meiermann | 57
# => [ { firstname: 'Harry', lastname: 'Langemann', age: 44 },
# { lastname: 'Heimstett', age: 32 }, { firstname: 'Bob', lastname: 'Meiermann', age: 57 } ]
Schema specifiers defined at first time use
Instead of defining a named schema specifiers at the very beginning of a GDS definition, you also can define it at the location of its first time use.
persons1 , @schema person( firstname, lastname, age )
: Harry | Langemann | 44
: Susi | Heimstett | 32
: Bob | Meiermann | 57
persons2 , @schema person
: Ludwig | Reinemann | 33
: Claudia | Schmidt | 67
# => { persons1: [ { firstname: 'Harry', lastname: 'Langemann', age: 44 }, { firstname: 'Susi', lastname: 'Heimstett', age: 32 },
# { firstname: 'Bob', lastname: 'Meiermann', age: 57 } ],
# persons2: [ { firstname: 'Ludwig', lastname: 'Reinemann', age: 33 }, { firstname: 'Claudia', lastname: 'Schmidt', age: 67 } ] }
Anonymous schema specifiers
If the schema specifier is used only once, there is no need to define an identifier for it, you can use an anonymous schema specifier.
persons , @schema( firstname, lastname, age )
: Harry | Langemann | 44
: Susi | Heimstett | 32
: Bob | Meiermann | 57
# => { persons: [ { firstname: 'Harry', lastname: 'Langemann', age: 44 }, { firstname: 'Susi', lastname: 'Heimstett', age: 32 },
# { firstname: 'Bob', lastname: 'Meiermann', age: 57 } ] }
Edge cases
If there are less values listed than keys for the schema specifier are defined, then for the resulting hash there will only key-value pairs for the available values be created.
, @schema( firstname, lastname, age )
: Harry | Langemann | 44
: Susi
: Bob | Meiermann | 57
# => [ { firstname: 'Harry', lastname: 'Langemann', age: 44 },
# { firstname: 'Susi' }, { firstname: 'Bob', lastname: 'Meiermann', age: 57 } ]
Just to mention, using the @na directive would lead to the same result, but is more verbose :
, @schema( firstname, lastname, age )
: Harry | Langemann | 44
: Susi | @na | @na
: Bob | Meiermann | 57
# => [ { firstname: 'Harry', lastname: 'Langemann', age: 44 },
# { firstname: 'Susi' }, { firstname: 'Bob', lastname: 'Meiermann', age: 57 } ]
If there are more values listed than keys for the schema specifier are defined, then an exception is raised.
, @schema( firstname, lastname, age )
: Harry | Langemann | 44
: Susi | Heimstett | 32 | 99
: Bob | Meiermann | 57
# Exception (SyntaxError) is raised
#
# GDS: too many values for schema definition: line 3, column 5
# --->
# , @schema( firstname, lastname, age )
# : Harry | Langemann | 44
# : Susi | Heimstett | 32 | 99
# ^
# <---
References
References can be used to define an alias for a certain sub structure, that means for a nested array or a nested hash.
Afterwards in the GDS definition, you can use the reference to refer to this sub structure.
For the definition of a reference you prefix an identifier (unique name) with an & ampersand.
For the use of a reference you prefix the same identifier with an * asterisk.
| &ref | define the reference named ref |
| *ref | use the reference named ref |
Let’s start with an example:
&ref config
param1 value 1 | param2 value 2
special *ref
# => { config: { param1: "value 1", param2: "value 2" }, special: { config: { param1: "value 1", param2: "value 2" } } }
References can be defined at miscellaneous places. Here are more examples:
config &ref
param1 value 1 | param2 value 2
special *ref
# => { config: { param1: "value 1", param2: "value 2" }, special: { param1: "value 1", param2: "value 2" } }
config
&ref param1 value 1 | param2 value 2
param3 value 3
special *ref
# => { config: { param1: "value 1", param2: "value 2", param3: "value 3" }, special: { param1: "value 1", param2: "value 2" } }
config
param1 value 1 | param2 value 2
&ref param3 value 3
special *ref
# => { config: { param1: "value 1", param2: "value 2", param3: "value 3" }, special: { param3: "value 3" } }
Examples for references in combination with arrays:
config
&ref list ,
1
2
3
special *ref
# => { config: { list: [ 1, 2, 3 ] }, special: { list: [ 1, 2, 3 ] } }
config
list &ref ,
1
2
3
special *ref
# => { config: { list: [ 1, 2, 3 ] }, special: [ 1, 2, 3 ] }
config
list , &ref 1 | 2
3
special *ref
# => { config: { list: [ 1, 2, 3 ] }, special: [ 1, 2 ] }
config
list , &ref 1
2
3
special *ref
# => { config: { list: [ 1, 2, 3 ] }, special: [ 1 ] }
config
list , 1
&ref 2 | 3
4
special *ref
# => { config: { list: [ 1, 2, 3, 4 ] }, special: [ 2, 3 ] }
And again more examples:
,
&ref,
1
2
3
4
*ref
# => [ [ 1, 2, 3, 4 ], [ 1, 2, 3, 4 ] ]
,
, &ref 1 | 2
3
4
*ref
# => [ [ 1, 2, 3, 4, [ 1, 2 ] ] ]
,
&ref k :
k1 v1
k2 v2
2
3
*ref
# => [ { k: { k1: "v1", k2: "v2" } }, 2, 3, { k: { k1: "v1", k2: "v2" } } ]
,
k &ref :
k1 v1
k2 v2
2
3
*ref
# => [ { k: { k1: "v1", k2: "v2" } }, 2, 3, { k1: "v1", k2: "v2" } ]
,
1
&ref :
k1
k11 v11
k2
k21 v21
2
3
*ref
# => [ 1, { k1: { k11: "v11" }, k2: { k21: "v21" } }, 2, 3, { k1: { k11: "v11" }, k2: { k21: "v21" } } ]
,
1
:
&ref k1
k11 v11
k2
k21 v21
2
3
*ref
# => [ 1, { k1: { k11: "v11" }, k2: { k21: "v21" } }, 2, 3, { k1: { k11: "v11" } } ]
If a reference is used which was not defined before, then an exception is raised.
Merging a hash into another - @merge
The @merge directive can be used to merge a hash specified with a reference into another hash.
An example similar to the Rails database configuration file config/database.yml
default &default
adapter sqlite3
pool 5
timeout 5000
development
@merge *default
database db/development.sqlite3
test
@merge *default
database db/test.sqlite3
production
@merge *default
database db/production.sqlite3
# => {
# default: { adapter: 'sqlite3', pool: 5, timeout: 5000 },
# development: { adapter: 'sqlite3', pool: 5, timeout: 5000, database: 'db/development.sqlite3' },
# test: { adapter: 'sqlite3', pool: 5, timeout: 5000, database: 'db/test.sqlite3' },
# production: { adapter: 'sqlite3', pool: 5, timeout: 5000, database: 'db/production.sqlite3' }
# }
After you have merged a hash, you can overwrite existing hash entries.
$logpath = /var/log/myapp
initial_settings &init
priority 10
max_size 2000
max_age 10
all_log_files ,
: @merge *init
file $(logpath)/logfile_1.log
: @merge *init
file $(logpath)/logfile_2.log
: @merge *init
file $(logpath)/logfile_3.log
max_age 8 # overwrite an existing hash entry
# => { initial_settings: { priority: 10, max_size: 2000, max_age: 10 },
# all_log_files: [
# { priority: 10, max_size: 2000, max_age: 10, file: "/var/log/myapp/logfile_1.log" },
# { priority: 10, max_size: 2000, max_age: 10, file: "/var/log/myapp/logfile_2.log" },
# { priority: 10, max_size: 2000, max_age: 8, file: "/var/log/myapp/logfile_3.log" } ] }
Inserting an array into another - @insert
The @insert directive can be used to insert an array specified with a reference into another array.
This is an example:
config_files &config_files ,
Gemfile
config/database.yml
config/application.gdstruct
docu_files &docu_files ,
README.md
CHANGELOG.md
License.md
app_files &app_files ,
appmodel.rb
appview.rb
appcontroller.rb
complete_file_list ,
@insert *config_files
@insert *docu_files
@insert *app_files
my_special.rb
# => { config_files: [ "Gemfile", "config/database.yml", "config/application.gdstruct" ],
# docu_files: [ "README.md", "CHANGELOG.md", "License.md" ],
# app_files: [ "appmodel.rb", "appview.rb", "appcontroller.rb" ],
# complete_file_list: [ "Gemfile", "config/database.yml", "config/application.gdstruct", "README.md", "CHANGELOG.md",
# "License.md", "appmodel.rb", "appview.rb", "appcontroller.rb", "my_special.rb" ] }
References together with Schema Specifiers
References are often used in the context of schema definitions to realize associations like one-to-one, one-to-many or many-to-many relationships.
The data structures can be defined in one block and easily be stored in a database.
The following example shows a list of bicycles with each of them assigned to a bike category.
There is a one-to-many relationship between category and bike. One category can be used for many bikes. More natural, many bikes can be assigned to one category.
categories , @schema( name )
&e_bike : E-Bike
&mountain_bike : Mountain Bike
&city_bike : City Bike
&kids_bike : Kids Bike
bikes , @schema( name, frame_size, wheel_diameter, weight, category ) /*
----------------------------------------------------------------------------------------
name frame size ["] wheel diameter ["] weight [kg] category
---------------------------------------------------------------------------------------- */
: Speedstar I | 18 | 27.5 | 23.3 | *city_bike
: Speedstar I | 20 | 27.5 | 24.4 | *city_bike
: Speedstar II | 22 | 27.5 | 25.5 | *city_bike
: Little Pony | 16 | 16 | 5.98 | *kids_bike
: Rocket | 20 | 20 | 7.13 | *kids_bike
: Easy Cruiser | 20 | 28 | 29.4 | *e_bike
: Rough Buffalo | 19 | 26 | 14.7 | *mountain_bike
# => { categories: [ { name: "E-Bike" }, { name: "Mountain Bike" }, { name: "City Bike" }, { name: "Kids Bike" } ],
# bikes: [ { name: "Speedstar I", frame_size: 18, wheel_diameter: 27.5, weight: 23.3, category: { name: "City Bike" } },
# { name: "Speedstar I", frame_size: 20, wheel_diameter: 27.5, weight: 24.4, category: { name: "City Bike" } },
# { name: "Speedstar II", frame_size: 22, wheel_diameter: 27.5, weight: 25.5, category: { name: "City Bike" } },
# { name: "Little Pony", frame_size: 16, wheel_diameter: 16, weight: 5.98, category: { name: "Kids Bike" } },
# { name: "Rocket", frame_size: 20, wheel_diameter: 20, weight: 7.13, category: { name: "Kids Bike" } },
# { name: "Easy Cruiser", frame_size: 20, wheel_diameter: 28, weight: 29.4, category: { name: "E-Bike" } },
# { name: "Rough Buffalo", frame_size: 19, wheel_diameter: 26, weight: 14.7, category: { name: "Mountain Bike" } } ] }
Please also take a look in the chapter Practical Examples where several examples for simple Rails applications with their database seeding files are shown.
Classic Ruby Syntax
The GDS language also supports the classic Ruby syntax for the notation of hash and array structures. You are using familiar curly braces and square brackets for structuring.
{ k1: 'v1', k2: 'v2' }
also this syntax is valid:
{ :k => {:k1=>'v1'} }
[ 1, 1_000, 2.0, 3.0e10, true, false, nil, { key1: 'val', key2: :"a complicated symbol &&%" } ]
[
1,
2.0,
true,
{
key1: "val",
key2: :"a complicated symbol &&%",
key3: [
"string1",
"string2"
]
}
]
There are a couple of peculiarities you have to consider:
- Keys of a hash always have to be a symbol (Ruby symbol).
- For values work integer literals, floating-point literals, symbols (Ruby symbols) and string literals as expected.
- The GDS default string literals can’t be used. You always have to quote string literals explicitly.
- Keyword literals have to be written like in Ruby, without a leading exclamation mark (!) : true, false, nil
- Schema specifiers can’t be used.
- The syntax is not indentation-sensitive.
- You have to decide right at the start, with the first expression, if you want to use the special, indentation-sensitive GDS language syntax or the classic Ruby syntax. You must not switch to the other syntax later or mix the two.
- You can use inline comments and block comments.
Why should I use the GDS language to define structures in the classic Ruby syntax?
Let’s say you develop a web application where you want to provide the user the possiblility to input hash/array structures.
Reasons for this could be for example to use these definitions as test data, as input for functions to convert into another format
or as expected results to verify in unit tests, which can be specified in this fictitious web application.
It would not be a good idea to evaluate those expressions with Ruby itself (eval(), instance_eval()), as this would cause code injection vulnerabilities,
or more specific dynamic evaluation vulnerabilities.
The proper way is to use a separate interpreter to evaluate these untrusted data.
If you don’t like the indentation-sensitive style provided by the GDS syntax, you can use the classic Ruby syntax and still benefit from the protection against code injection vulnerabilities.
Language Extensions
The GDS language provides a couple of special directives. Each directive begins with a @ character.
Accessing environment variables - @env
The @env directive can be used to access environment variables.
You can use it as a special kind of value inside an array or as the value part of a key-value pair inside a hash.
You have to name the environment variable within parentheses.
For configuration files you can supply properties with environment variables.
Environment variables enable you to parameterize configuration files and dynamically pass settings to them, without changing the main configuration file. By separating these settings from the configuration file, you don’t need to update your configuration file when you need to change the configuration based on different settings. This situation frequently happens, when your application is going through different lifecycle stages like development, testing and production. Each stage uses its own environment, for example its own database.
Example configuration:
# configuration file
# file name: /develop/projectconf.gdstruct
project_settings
superuser Administrator
project @env(MYPROJECT_NAME)
projectid @env(MYPROJECT_ID)
database @env(MYPROJECT_DATABASE)
db_adapter @env(MYPROJECT_DB_ADAPTER)
d = GDstruct.create_from_file( '/develop/projectconf.gdstruct', allow_env: true )
# => {:project_settings=>{:superuser=>"Administrator", :project=>"MyProject", :projectid=>"1001", :database=>"specialdb", :db_adapter=>"postgres"}}
This example uses a separate file for the GDS definition. The class method create_from_file (alias c_from_file) interprets the GDS definition.
To prevent security vulnerabilities and to make sure that you use this feature only in a controlled context,
you have to set the optional parameter allow_env to true, when you want to use the @env directive to access environment variables.
This works the same for the class method create (alias c).
Evaluating embedded Ruby code - @r
The @r directive (r for Ruby) can be used to evaluate embedded Ruby source code.
You can use it as a special kind of value inside an array or as the value part of a key-value pair inside a hash.
You have to note the Ruby code within parentheses.
Example with embedded Ruby code:
class MyClass
attr_accessor :a, :b
def initialize
@a = 80
@b = 90
end
end
myobj = MyClass.new
@myhash = { k1: "v1", k2: "v2" }
d = GDstruct.c( <<EOS, context: binding )
,
@r( (1..10).to_a )
@r( myobj ) # local variable
@r( @myhash ) # instance variable
@r(
o = MyClass.new
o.a = 50**2
o.b = 4000
o
)
: year @r(Time.now.year)
EOS
# => [[1, 2, 3, 4, 5, 6, 7, 8, 9, 10], #<MyClass:0x3afc248 @a=80, @b=90>, {:k1=>"v1", :k2=>"v2"}, #<MyClass:0x3a34cb8 @a=2500, @b=4000>, {:year=>2018}]
To prevent security vulnerabilities and to make sure that you use this feature only in a controlled environment, you have to set the optional parameter context to the binding in which you want to execute the Ruby code. This works the same for the class method create_from_file (alias c_from_file).
Limitations
There are several items on the TODO list to make the GDS language more expressive and more flexible.
Please let me know when you hit some limiting borders.
Example: Conversion to JSON
You can use the GDS language to define a data structure in a clear and succinct representation and after that convert it to the JSON data format.
Example:
require 'gdstruct'
require 'json'
res = GDstruct.c( <<-EOS ).to_json
caption foo
credit bar
images
small
url http://mywebsite.com/image-small.jpg
dimensions
height 500
width 500
large
url http://mywebsite.com/image-large.jpg
dimensions
height 500
width 500
videos
small
preview http://mywebsite.com/video.m4v
dimensions
height 300
width 400
EOS
The JSON representation:
{"caption":"foo","credit":"bar","images":{"small":{"url":"http://mywebsite.com/image-small.jpg","dimensions":{"height":500,"width":500}},"large":{"url":"http://mywebsite.com/image-large.jpg","dimensions":{"height":500,"width":500}}},"videos":{"small":{"preview":"http://mywebsite.com/video.m4v","dimensions":{"height":300,"width":400}}}}
Prettier formated the JSON representation looks the following:
{ "caption": "foo",
"credit": "bar",
"images": {
"small": {
"url": "http://mywebsite.com/image-small.jpg",
"dimensions": {
"height": 500,
"width": 500 } },
"large": {
"url": "http://mywebsite.com/image-large.jpg",
"dimensions": {
"height": 500,
"width": 500 } } },
"videos": {
"small": {
"preview": "http://mywebsite.com/video.m4v",
"dimensions": {
"height": 300,
"width": 400 } } } }
Example: Conversion to XML
You can use the GDS language to define a data structure in a clear and succinct representation and after that convert it to the XML data format.
Example:
require 'gdstruct'
require 'active_support/core_ext/array/conversions'
require 'active_support/core_ext/hash/conversions'
res = GDstruct.c( <<-EOS ).to_xml(root: :data)
caption foo
credit bar
images
small
url http://mywebsite.com/image-small.jpg
dimensions
height 500
width 500
large
url http://mywebsite.com/image-large.jpg
dimensions
height 500
width 500
videos
small
preview http://mywebsite.com/video.m4v
dimensions
height 300
width 400
EOS
The XML representation looks the following:
<?xml version="1.0" encoding="UTF-8"?>
<data>
<caption>foo</caption>
<credit>bar</credit>
<images>
<small>
<url>http://mywebsite.com/image-small.jpg</url>
<dimensions>
<height type="integer">500</height>
<width type="integer">500</width>
</dimensions>
</small>
<large>
<url>http://mywebsite.com/image-large.jpg</url>
<dimensions>
<height type="integer">500</height>
<width type="integer">500</width>
</dimensions>
</large>
</images>
<videos>
<small>
<preview>http://mywebsite.com/video.m4v</preview>
<dimensions>
<height type="integer">300</height>
<width type="integer">400</width>
</dimensions>
</small>
</videos>
</data>
Comparing to YAML
You can use the GDS language to define a data structure in a clear and succinct representation and after that convert it to the YAML data format.
Example:
require 'gdstruct'
require 'yaml'
res = GDstruct.c( <<-EOS ).to_yaml
caption foo
credit bar
images
small
url http://mywebsite.com/image-small.jpg
dimensions
height 500
width 500
large
url http://mywebsite.com/image-large.jpg
dimensions
height 500
width 500
videos
small
preview http://mywebsite.com/video.m4v
dimensions
height 300
width 400
EOS
The YAML representation looks the following:
---
:caption: foo
:credit: bar
:images:
:small:
:url: http://mywebsite.com/image-small.jpg
:dimensions:
:height: 500
:width: 500
:large:
:url: http://mywebsite.com/image-large.jpg
:dimensions:
:height: 500
:width: 500
:videos:
:small:
:preview: http://mywebsite.com/video.m4v
:dimensions:
:height: 300
:width: 400
As you see, the YAML representation is also using an indentation-sensitive style and in general does look very similar. Mostly there are a couple of additional colons (:) in YAML.
Significant advantages of GDS compared to YAML are schema specifiers and the definition of multiple values and multiple key-value pairs in one line, using the vertical bar symbol (|) .
require 'gdstruct'
require 'yaml'
res = GDstruct.c( <<-EOS ).to_yaml
@schema country(name,capital,area,population,vehicleRegistrationCode,iso3166code,callingCode)
, @schema country /*
----------------------------------------------------------------------------------------------------------------------------
name capital area (km^2) population vehicleRegistrationCode iso3166code callingCode
---------------------------------------------------------------------------------------------------------------------------- */
: Deutschland | Berlin | 357_385 | 82_521_653 | D | DE | 49
: USA | Washington, D.C. | 9_833_520 | 325_719_178 | USA | US | 1
: China | Beijing | 9_596_961 | 1_403_500_365 | CHN | CN | 86
: India | New Dehli | 3_287_469 | 1_339_180_000 | IND | IN | 91
: Austria | Vienna | 83_878 | 8_822_267 | A | AT | 43
: Denmark | Copenhagen | 42_921 | 5_748_769 | DK | DK | 45
: Canada | Ottawa | 9_984_670 | 36_503_097 | CDN | CA | 1
: France | Paris | 643_801 | 66_991_000 | F | FR | 33
: Russia | Moscow | 17_075_400 | 144_526_636 | RUS | RU | 7
EOS
The equivalent YAML representation looks the following:
---
- :name: Deutschland
:capital: Berlin
:area: 357385
:population: 82521653
:vehicleRegistrationCode: D
:iso3166code: DE
:callingCode: 49
- :name: USA
:capital: Washington, D.C.
:area: 9833520
:population: 325719178
:vehicleRegistrationCode: USA
:iso3166code: US
:callingCode: 1
- :name: China
:capital: Beijing
:area: 9596961
:population: 1403500365
:vehicleRegistrationCode: CHN
:iso3166code: CN
:callingCode: 86
- :name: India
:capital: New Dehli
:area: 3287469
:population: 1339180000
:vehicleRegistrationCode: IND
:iso3166code: IN
:callingCode: 91
- :name: Austria
:capital: Vienna
:area: 83878
:population: 8822267
:vehicleRegistrationCode: A
:iso3166code: AT
:callingCode: 43
- :name: Denmark
:capital: Copenhagen
:area: 42921
:population: 5748769
:vehicleRegistrationCode: DK
:iso3166code: DK
:callingCode: 45
- :name: Canada
:capital: Ottawa
:area: 9984670
:population: 36503097
:vehicleRegistrationCode: CDN
:iso3166code: CA
:callingCode: 1
- :name: France
:capital: Paris
:area: 643801
:population: 66991000
:vehicleRegistrationCode: F
:iso3166code: FR
:callingCode: 33
- :name: Russia
:capital: Moscow
:area: 17075400
:population: 144526636
:vehicleRegistrationCode: RUS
:iso3166code: RU
:callingCode: 7
Here, in my opinion, the GDS representation is clearer.
Practical Examples
There is a collection of practical examples on GitHub:
github.com/uliramminger/gds-examples
The following are a couple examples of Rails applications using the GDS language to seed initial data into the database.
Countries01
This example includes only one Active Record model: Country
The file to seed initial data into the database (db/seeds.rb) looks the following
# db/seeds.rb
countries = GDstruct.c( <<-EOS )
@schema country( name, capital, area, population, vehicleRegistrationCode, iso3166code, callingCode )
, @schema country /*
----------------------------------------------------------------------------------------------------------------------------
name capital area (km^2) population vehicleRegistrationCode iso3166code callingCode
---------------------------------------------------------------------------------------------------------------------------- */
: Australia | Canberra | 7_692_024 | 25_130_600 | AUS | AU | 61
: Austria | Vienna | 83_878 | 8_822_267 | A | AT | 43
: Belgium | Brussels | 30_528 | 11_420_163 | B | BE | 32
: Brazil | Brazília | 8_515_767 | 210_147_125 | BR | BR | 55
: Canada | Ottawa | 9_984_670 | 36_503_097 | CDN | CA | 1
: China | Beijing | 9_596_961 | 1_403_500_365 | CHN | CN | 86
: Costa Rica | San José | 51_100 | 4_857_274 | CR | CR | 506
: Denmark | Copenhagen | 42_921 | 5_748_769 | DK | DK | 45
: Deutschland | Berlin | 357_385 | 82_521_653 | D | DE | 49
: Estonia | Tallinn | 45_227 | 1_319_133 | EST | EE | 372
: France | Paris | 643_801 | 66_991_000 | F | FR | 33
: India | New Dehli | 3_287_469 | 1_339_180_000 | IND | IN | 91
: Mauritius | Port Louis | 2_040 | 1_262_132 | MS | MU | 230
: Niger | Niamey | 1_267_000 | 20_672_987 | RN | NE | 227
: Nigeria | Abuja | 923_768 | 190_886_311 | WAN | NG | 234
: Russia | Moscow | 17_075_400 | 144_526_636 | RUS | RU | 7
: USA | Washington, D.C. | 9_833_520 | 325_719_178 | USA | US | 1
EOS
countries.each do |c|
country = Country.create!( c )
end
Countries02
This example includes two Active Record models: Country and City.
There is a one-to-many relationship between Country and City. One country has many cities.
The file to seed initial data into the database (db/seeds.rb) looks the following
# db/seeds.rb
countries = GDstruct.c( <<-EOS )
@schema country( name, capital, area, population, vehicleRegistrationCode, iso3166code, callingCode )
@schema city( name, population )
, @schema country /*
--------------------------------------------------------------------------------------------------------------------------------------------------------------
name capital area (km^2) population vehicleRegistrationCode iso3166code callingCode || name population
-------------------------------------------------------------------------------------------------------------------------------------------------------------- */
: Australia | Canberra | 7_692_024 | 25_130_600 | AUS | AU | 61
bigCities , @schema city
: Sydney | 5_131_326
: Melbourne | 4_850_740
: Brisbane | 2_408_223
: Austria | Vienna | 83_878 | 8_822_267 | A | AT | 43
bigCities , @schema city
: Vienna | 1_840_573
: Graz | 272_838
: Linz | 198_181
: Salzburg | 148_420
: Innsbruck | 126_851
: Brazil | Brazília | 8_515_767 | 210_147_125 | BR | BR | 55
: Canada | Ottawa | 9_984_670 | 36_503_097 | CDN | CA | 1
bigCities , @schema city
: Toronto | 2_731_571
: China | Beijing | 9_596_961 | 1_403_500_365 | CHN | CN | 86
bigCities , @schema city
: Chongqing | 30_165_500
: Shanghai | 24_183_300
: Beijing | 21_707_000
: Guangzhou | 13_081_000
: Tianjin | 11_249_000
: France | Paris | 643_801 | 66_991_000 | F | FR | 33
bigCities , @schema city
: Paris | 2_229_621
: Marseille | 855_393
: Mauritius | Port Louis | 2_040 | 1_262_132 | MS | MU | 230
: Niger | Niamey | 1_267_000 | 20_672_987 | RN | NE | 227
: Russia | Moscow | 17_075_400 | 144_526_636 | RUS | RU | 7
bigCities , @schema city
: Moscow | 12_380_664
: Saint Petersburg | 5_281_579
: Novosibirsk | 1_602_915
: Yekaterinburg | 1_455_514
: USA | Washington, D.C. | 9_833_520 | 325_719_178 | USA | US | 1
EOS
countries.each do |countrydef|
country = Country.create!( countrydef.except(:bigCities) )
country.bigCities.create!( countrydef[:bigCities] ) if countrydef[:bigCities]
end
Countries03
This example includes four Active Record models: Country, City, Museum and Person.
There are a couple of one-to-many relationships:
- between Country and City: one country has many cities
- between City and Museum: one city has many museums
- between Country and Person: one country has many famous people
The file to seed initial data into the database (db/seeds.rb) looks the following
# db/seeds.rb
countries = GDstruct.c( <<-EOS )
@schema country( name, capital, area, population, vehicleRegistrationCode, iso3166code, callingCode )
@schema city( name, population )
@schema museum( name, established )
@schema person( firstname, lastname, yearOfBirth )
, @schema country /*
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Country || City || Museum || Person
name capital area (km^2) population vehicleRegistrationCode iso3166code callingCode || name population || name | established (year) || firstname | lastname | year of birth
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- */
: Austria | Vienna | 83_878 | 8_822_267 | A | AT | 43
bigCities , @schema city
: Vienna | 1_840_573
: Graz | 272_838
: Linz | 198_181
famousPeople, @schema person
: Wolfgang Amadeus | Mozart | 1756
: Kurt | Gödel | 1906
: Deutschland | Berlin | 357_385 | 82_521_653 | D | DE | 49
bigCities , @schema city
: Berlin | 3_613_495
museums, @schema museum
: Pergamon Museum | 1910
: Bode Museum | 1904
: Bauhaus Archive | 1960
: Natural History Museum | 1889
: Munich | 1_456_039
museums, @schema museum
: Deutsches Museum | 1903
famousPeople, @schema person
: Albert | Einstein | 1879
: Friedrich | Schiller | 1759
: Johann Sebastian | Bach | 1685
: France | Paris | 643_801 | 66_991_000 | F | FR | 33
bigCities , @schema city
: Paris | 2_229_621
: Marseille | 855_393
: USA | Washington, D.C. | 9_833_520 | 325_719_178 | USA | US | 1
EOS
countries.each do |countrydef|
country = Country.create!( countrydef.except(:bigCities,:famousPeople) )
countrydef[:bigCities].each do |citydef|
city = City.new( citydef.except(:museums) )
country.bigCities << city
city.museums.create!( citydef[:museums] ) if citydef[:museums]
end if countrydef[:bigCities]
country.famousPeople.create!( countrydef[:famousPeople] ) if countrydef[:famousPeople]
end
Countries04
This example includes two Active Record models: Currency and Country.
There is a one-to-many relationship between Currency and Country. One currency is used in (has many) countries.
This time there are references used to establish the relationship between Country and Currency.
The file to seed initial data into the database (db/seeds.rb) looks the following
# db/seeds.rb
data = GDstruct.c( <<-EOS )
currencies , @schema( name, symbol, iso4217code, iso4217number, subunit ) /*
------------------------------------------------------------------------------------------------------------
name symbol iso4217code iso4217number subunit
------------------------------------------------------------------------------------------------------------ */
&australian_dollar : Australian dollar | '$' | AUD | 36 | cent
&brazilian_real : Brazilian real | 'R$' | BRL | 986 | centavo
&euro : Euro | '€' | EUR | 978 | cent
&united_states_dollar : United States dollar | '$' | USD | 840 | cent
countries , @schema( name, capital, area, population, currency ) /*
-----------------------------------------------------------------------------------------------------------
name capital area (km^2) population currency
----------------------------------------------------------------------------------------------------------- */
: Australia | Canberra | 7_692_024 | 25_130_600 | *australian_dollar
: Austria | Vienna | 83_878 | 8_822_267 | *euro
: Brazil | Brasília | 8_515_767 | 210_147_125 | *brazilian_real
: British Virgin Islands | Road Town | 153 | 31_758 | *united_states_dollar
: Christmas Island | Flying Fish Cove | 135 | 1_843 | *australian_dollar
: Deutschland | Berlin | 357_385 | 82_521_653 | *euro
: France | Paris | 643_801 | 66_991_000 | *euro
: Guam | Hagåtña | 540 | 162_742 | *united_states_dollar
: Kiribati | Tarawa | 811 | 110_136 | *australian_dollar
: Marshall Islands | Majuro | 181 | 53_066 | *united_states_dollar
: Nauru | Yaren | 21 | 11_200 | *australian_dollar
: Palau | Melekeok | 459 | 21_503 | *united_states_dollar
: Puerto Rico | San Juan | 9_104 | 3_337_177 | *united_states_dollar
: USA | Washington, D.C. | 9_833_520 | 325_719_178 | *united_states_dollar
EOS
data[:currencies].each do |currencydef|
currency = Currency.create!( currencydef )
currencydef[:id] = currency.id
end
data[:countries].each do |countrydef|
Country.create!( countrydef.merge( currency_id: countrydef[:currency][:id] ).except(:currency) )
end
Books01
This example includes three Active Record models: Book, Author and Publication.
There is a many-to-many relationship between Book and Author.
One book can be written by many (has many) authors and one author can write (has many) books.
The Publication model serves as a join model to setup the many-to-many relationship.
There are references used to establish the relationship between Book and Author.
The file to seed initial data into the database (db/seeds.rb) looks the following
# db/seeds.rb
data = GDstruct.c( <<-EOS )
authors , @schema( firstname, lastname ) /*
----------------------------------------------------------
first name last name
---------------------------------------------------------- */
&h_abelson : Harold | Abelson
&e_gamma : Erich | Gamma
&j_goerzen : John | Goerzen
&r_helm : Richard | Helm
&r_johnson : Ralph | Johnson
&dr_musser : David R. | Musser
&b_o_sullivan : Bryan | O'Sullivan
&r_olsen : Russ | Olsen
&a_saini : Atul | Saini
&d_stewart : Don | Stewart
&gj_sussman : Gerald Jay | Sussman
&j_sussman : Julie | Sussman
&j_vlissides : John | Vlissides
books , @schema book( title, subtitle, year, isbn ) /*
-------------------------------------------------------------------------------------------------------------------------------------------
title subtitle year ISBN number
------------------------------------------------------------------------------------------------------------------------------------------- */
: Design Patterns | Elements of Reusable Object-Oriented Software | 1995 | '0-201-63361-2'
authors, *e_gamma | *r_helm | *r_johnson | *j_vlissides
: Design Patterns in Ruby | @na | 2008 | '0-321-49045-2'
authors, *r_olsen
: Real World Haskell | @na | 2009 | '0-596-51498-3'
authors, *b_o_sullivan | *j_goerzen | *d_stewart
: STL Tutorial and Reference Guide | C++ Programming with the Standard Template Library | 1996 | '0-201-63398-1'
authors, *dr_musser | *a_saini
: Structure and Interpretation of Computer Programs | Second Edition | 1996 | '0-262-51087-1'
authors, *h_abelson | *gj_sussman | *j_sussman
EOS
data[:authors].each do |authordef|
author = Author.create!( authordef )
authordef[:id] = author.id
end
data[:books].each do |bookdef|
book = Book.create!( bookdef.except(:authors) )
bookdef[:authors].each do |authordef|
book.publications.create!( book_id: book.id, author_id: authordef[:id] )
end
end
Implementation
For the parsing of GDS expressions the treetop gem is used.
For the creation of the treetop’s grammar definition for the GDS language,
a tool called LDL (Language Definition Language) generator is used.
The LDL metalanguage is a special language for the definition of other languages
and provides a higher abstraction for language definitions and the creation of abstract syntax trees.
The LDL generator is homemade and is not published yet.
Ruby Gem
You can find it on RubyGems.org:
Source Code
You can find the source code on GitHub: