Dataset lifecycle¶

Datasets follow a standard life cycle path, regardless on their nature and typology.

• Dataset entity creation: a dataset entity is created via a metadata management form, where the user specifies information about the dataset (following the required items of the DCATAP_IT standard), its data structure, properties and annotations of its fields/attributes, as well as operational information to govern the management of dataset in the platform (where to store the dataset, where to be listening for new data coming, who can have access to it, etc.).
• Ingestion: after the dataset entity has been created, a microservice activates an Apache NiFi pipeline that is listening for new data to ingest.DAF is currently ready to ingest data coming from SFTP (default option for batch data), pull and push from an external webservice.
• Transformation and enrichment pipelines: before being stored into the appropriate storage engine, the data goes through several pipelines that add information to the incoming data. We are currently developing the following two pipelines:
  • Normalization pipeline, to apply DAF internal conventions to raw incoming data, such as format (UTF-8), management of null entries, refactoring of codified fields like date and URL, and so on.
  • Standardization pipeline, to make sure that fields marked as bound to a controlled vocabulary (via semantic annotation made during the dataset entity creation phase) are actually using the terms present in the vocabulary.

All pipelines are thought to enrich the incoming raw data, so not to modify the original content: the steps described above add new fields with the result of the transformation applied and, when applicable, an information about the “goodness” of the transformation applied.

• Storage: after the dataset is ready, it will be persisted using one or more storage mechanism indicated during the dataset entity creation phase. We are currently working to support parquet files in HDFS, HBase, Kudu, MongoDB and ElasticSearch.
• E-gestion: data are consumed by the end user via API (implemented within the dataset manager microservice), Spark (accessible via Livy and a Jupyter notebook) and data analytics & visualization application (currently we integrated with Metabase and Superset).

Types of datasets¶

From a logical point of view, the DAF manages two types of datasets:

• Standard datasets describe phenomena that are common nationwide. These datasets are thus defined as datasets with national relevance and supported by the highest level of information/metadata. They follow a detailed set of rules and standardization mechanisms that make them homogeneous across data sources (there may be multiple data sources describing the same phenomena, e.g., the bike sharing phenomena can be analyzed using data coming from Milan, Turin, Rome, etc…).
• Ordinary datasets have “owner” relevance, in the sense that they are defined and generated by a specific owner for its specific usage. They do not obey to a standard nationwide schema (data model), but the owner needs to specify metadata and information about the dataset before ingesting the data into the Big Data Platform of the DAF.

Metadata, Semantic Tagging and Controlled Vocabularies¶

Datasets are associated to concepts and attributes described into domain ontologies via semantic tagging (done during the ingestion step). This mechanism consists in linking columns/features of the datasets to corresponding ontologies. This is done to specify the concepts and the attributes the datasets contain.

For example, let’s consider a dataset containing the list of Italian companies and their headquarters. Here all the columns related to personal information (e.g. name, year of establishment, etc.) will be tagged with the corresponding semantic tag contained in the “Organizations ontology”. Furthermore, the information related to the address of the headquarter will be tagged with the semantic tag of the address of from the “Places Ontology”, and also associated to the concept of “company”. In this way the address is the one of a company (as opposed, for example, to the address of a citizen). Finally, by having linked the address to the related concepts contained in the “Places ontology”, we will be able to use all info contained in the mentioned ontology, included the existence of controlled vocabularies for the properties used.

The advantage of using ontologies and linking them to the ingested dataset are the following:

• easier establishment and implementation of a standard dataset at national level;
• possibility to logically link datasets based on their content and concepts. This feature is specifically useful in exploratory analysis and model munging;
• usage of the controlled vocabulary information so to provide standardization procedures to ensure the correct usage of the vocabulary in the dataset, and therefore, improve interoperability;
• possibility to build meaningful and rich knowledge graphs to be exploited for further analysis.

Dataportal¶

The Dataportal is the DAF user interface, a place where users can have access to most of DAF functionalities and contents, based on their role and access grants. It is a web applications offering a public area where users can have access to contents that can be released publicly, and a private area where authenticated users can manage and use all data they have been granted access to based on their role and organization, and functionalities like data visualization, business intelligence and data science.

Here are some basic content and features accessed and managed via the dataportal:

• Dataset metacatalog: a catalog where to search for datasets, their info according to DCAPAP_IT profile, correlated datasets, related information and datastories made by using the dataset, the list of API with which using the dataset, and other operational info useful to use the dataset with the integrated tools.
• Data stories: blog posts that integrate data visualizations and scripts to tell the whys and the results of a data analysis. Data stories are design to stimulate discussion around an data related themes, and allows the community to comment an propose further analysis.
• Data applications: software with a user interfaces that expose functionalities based on data and models trained on them. Data applications can be done by the DAF team, public administration or can be proposed by the community, and will be organized in a related section of the dataportal.
• News & blog: blog post containing news related to the data world, together with how-to and tutorial on how to use the DAF and dataportal.
• Data visualization & Business Intelligence: we integrated two open source tools, Metabase and Superset, where user can explore data managed by DAF and create graphs and dashboards.
• Data science: a JupyterHub notebook has been integrated and connected to the Hadoop cluster to allow for data manipulation, analysis and data science. Users can take advantage of the capability of Apache Spark and its libraries to run SQL queries on top of Hadoop, integrate different datasets and training machine learning models using MLlib library.

API¶

The DAF exposes automatically an API to interact with datasets. Users can use standardized interfaces to access the datasets to:

• get statistical and metadata info;
• download the dataset (up to a certain limits);
• run SQL like queries on datasets.

The use of APIs will allow users to build their applications easily and with the guarantee to access the most updated data available.

Ordinary Dataset¶

Every datasets that are ingested by an organization (PA) are treated as ordinary dataset. An ordinary dataset does not have a pre-defined structure it needs to follow, it is ingested as it comes, besides the standardization, normalization and enrichment processes defined in the ingestion pipeline.

It will have the following data structure:

• standardized & normalized original columns
• raw original columns
• enrichment columns, among which (most of them will depend on the content of the original columns):
  • __ROWID: global unique row id
  • __dtcreated: date and time when the info has been added
  • __dtupdated: date and time when the info has been updated

Ordinary datasets can be created from data coming from outside DAF, or by transformations applied to already existing DAF datasets. For example, a public administration can decide to create an open data version of a private dataset, by indicating the columns that can be released publicly and an optional aggregation policy. In this way, everytime the private dataset receives updates, they will be automatically be reflected into its open data version (that will be another dataset in the DAF world).

Standard Datasets¶

Standard Datasets are those datasets that describe concepts and phenomena valid nationwide, following a strict data structure and semantics rules. Standard datasets are said to follow ‘standards’ that are defined for all PAs, so to guarantee that a given phenomena can be described in the same way and following the same conventions nationwide. Data will be physically stored in the following HDFS meta-directory:

They will typically have the following data structure:

• standardized and normalized mandatory columns, grouped under the mand.{colname} struct field
• standardized and normalized optional columns, grouped under the opt.{colname} struct field
• enrichment columns from the ingestion procedures, grouped under the enr.{colname} struct field
• operational enrichment columns, with info needed for internal DAF operations and grouped under the ops.{colname} struct fields. These fields are:
  • __dtcreated: date time when the info has been ingested into the standard dataset
  • __dtupdated: date time when the info has been updated
  • __srcorg: the code name of the organization that originated the data (e.g. ‘Comune_Milano’)
  • __dsname: the unique name of the ordinary dataset from which the data come from.

Dataset standards can be created directly by a PA, if it is the only contributor to the national standard, or by 2 or more PAs in cases when every PA contribute to the national standard with the piece of info it manages. In the latter case, the standard dataset is created starting from 2 or more ordinary datasets, and in this case can be considered a ‘derived’ dataset. In this case, the ordinary dataset will specify that they contribute to a standard and the mapping model needed to map the ordinary dataset into the standard one.

Raw Open Data¶

Finally, we collect and store raw open data coming from the national catalog. They are automatically metadated based on the info available (in ‘at best’ fashion), and stored in HDFS in the following path:

/daf/opendata/{organization}/{dataset name}

Once ingested, they can be used with all other tools managed by DAF.

Dataset level metadata (DCATAP_IT)¶

These metadata are meant to describe the dataset’s info such as its name, owner, category, etc. DAF implemented the mandatory fields of the DCATAP_IT profile, according to AGID regulations. Following a subset of the DCATAPIT metadata (see the link below for a complete list of required info)

• name: dataset unique name
• title: title of the dataset
• identifier: dataset unique identifier
• alternate_identifier:…
• author: …
• theme: thematic domain that characterize the dataset
• license_id:…
• resources: a list of resources from where to download the dataset and other related data. Dataset managed in DAF will have here API endpoints to download and access the dataset.
• license_title:
• frequency:
• publisher_name: name of the organization that publish the dataset
• publisher_identifier….
• organization:….
• owner_name: name of the organization that owns the dataset
• holder_name:…
• holder_identifier:…
• tags: tagging system connected to vocabulary.
• relationshiops_as_subject:…
• notes: additional information
• modified: date of last modification to the dataset

Data structure level metadata¶

Data scructure metadata contains information about the internal structure of a dataset, at column or field level. It manages info about format, content, semantics that are contained into a single column of a tabular dataset, for example. Here we store two datastructure metadata: an avro schema of the dataset, and a ‘flatschema’ where we store additional information than the one provided by the avro schema, specifically thought to enrich the information expressed about the content and the semantics of the columns. Below, you’ll find the list of info contained in the ‘flatschema’ part, we’ll skip the avro schema part as are using the standard avro schema definition.

• name: name of the field/column. This name needs to obey to formatting rules that are necessary for it to be used as the name of a column in a database, therefore it may not be human readable.

• type: data format of the column, such as ‘string’.

• metadata.title: human readable name for the column.

• metadata.desc: description of the content of the column.

• metadata.field_type: it tells if the column is a dimension, a metric (numeric attribute) or a descriptive attribute.

• metadata.required: it tells if the field is mandatory or optional.

• metadata.uniq_dim: checked if the column is part of the list of dimensions that make the row unique, such that there will not be two rows with the same values for the columns checked as uniq_dim.

• metadata.is_createdate: boolean, checked if the column contains the date when the row was created.

• metadata.is_updatedate: boolean, checked if the column contains the date when the row was updated.

• metadata.cat: category that can better represent the content of the field. This is controlled by a vocabulary.

• metadata.tag: list of tags that can better represent the content of the field. All tags are saved into an evolving vocabulary.

• metadata.constr: a list of objects (made by type and param arguments) to set contraints on the content of the field.

• metadata.semantics: an object containing semantic info to link hte column to related ontology and controlled vocabulary, if any. In particular:

  • id: this is the semantic tag that links the column with a given attribute of a concept described into an ontology.
  • context: this info gives context info on the semantic tag.
  • rdf_subject: it is used to give a better context to the info contained in the column. Technically, it is a tag for a concept described into an ontology. In most cases, it can be seen as the subject that makes an action, derived from the id attribute.
  • rdf_predicate: the action that the subject perform on the content of the column.
  • rdf_object: the target of the action performed by the subject.
  • uri_voc: It is a unique identifier for the vocabulary. It matches with the dsname field of the dataset in DAF.
  • uri_property: it is the uri associated to the element epressed in the column. It is used, among other things, to link the column of the dataset to the column of the vocabulary which controls it, if any.
  • property_hierarchy: it is of type array, and it gives info about the hierarchy, if any, to which the property/column belongs to.

• metadata.personal: this objects contains info whether the data are of personal kind. In particular:

  • ispersonal: boolean, to tell whether or not the info contained in the column is a personaltype of information.
  • cat: category of personal information

• metadata.format_std: this is an object that gives info about a format standard the data follows when ingested in DAF. It is useful to help the system identify such standard and transform into the DAF choosen standard. The object has the following two attributes:

  • name: name of the format standard, e.g. date, credit card, address, etc.
  • param: depending on the type of format, it is a string giving the exact formatting order and composition of the information contained. E.g. for the date example, it may be ‘YY/MM/DD’. This will help the normalization procedure to refactor in the right order and format the information, such to follow the DAF internal conventions.

• metadata.field: it has info on indexing in SearchEngine and profiling of the field, plus other Kylo specific information on standard and validation.

  • is_index: it tells to create an index based on this field in the SearchEngine.
  • is_profile: it tells to create a profile for the field that will be displayed as result of the SearchEngine.
  • validation: contains info on the validation rules to be used for the field.
  • standardization: contains info on the standardization procedure to be performed on the field.

  Operational level metadata

These metadata are used to manage the dataset within DAF logics and conventions, from input sources to storage options to ingesiton pipelines mechanics.

• inactive: optional boolean, true if the dataset entry has been created as inactive (that is, no effects on the system has been created, e.g. no ingestion pipeline has been started for the dataset yet).
• theme:..
• subtheme:…
• logical_uri:
• physical_uri:..
• is_std:..
• group_own:…
• group_access: (name, role)
• std_schema:…
• georef:..
• input_src: It is an object containing information about input feeds, which can be of the following types:
  • sftp: it contains info on how to access the sftp plus specific info on the feed characteristics (i.e. data format and related options) * name * url * username * password * param: this is a json string containing info related to the specific input type not already codified. An important info contained here is the type of file to be ingested (e.g. csv, json, xml) and option related to the file format.
  • srv_pull: , srv_push, daf_dataset. Each of them,
• ingestion_pipeline:..
• storage_info: (hdfs, kudu, hbase, mongo, textdb)
• dataset_proc: It has info about how to process and store internally the dataset. Such info includes partitioning, merge strategy, etc.
  • read_type: update vs timeseries
  • dataset_type: batch vs stream
  • partitions: It contains info on how the dataset is partitioned in DAF.
    • name: name of the partition, given by the user.
    • field: name of the field to be used for partitioning. It must correspond to one of the ‘name’ of the dataschema.
    • formula: the formula to be applied to the field to get the partition value.
  • merge_strategy: It tells how new data should be ingested into the existing dataset. User must choose among the following options. ‘SYNC’ to replace the existing content with the new one; ‘MERGE’ to append the data into the target partitions; ‘DEDUPE_AND_MERGE’ to insert into the target partition but ensure no duplicate rows are remaining; ‘PK_MERGE’ to insert or update existing rows matching the same primary key; ‘ROLLING_SYNC’ to overwrite target partitions only when present in source.
• opendata: it is used to tell the system to create an open data version of the dataset. If valued, it will create a new derived dataset entry, precompiled with info taken from the original dataset, and put into ‘inactive’ state so it can be valued and confirmed by the user. It is an object with the following info: * create_opendata: boolean, valued as true if user wants to create a derived open data dataset. * sql: SQL query with the final data structure of the open data dataset.
• service_layer: it is used to put the dataset (or its transformation) into the service layer (Kudu?) * transfer_mode: it tells whether the dataset will be put as is in the service layer or it needs to be transformed via derived dataset. It takes two values: direct, derived. * sql: optional, used in case transfer_mode is valued at derived and user wants to specify ex-ante the transformation query.

DAF Conventions List¶

• Data format (Normalization): all data is transformed into UFT-8 format
• Data conventions (Normalization): we make use of the following internal conventions
  • null value are saved as NULL. Therefore, all other external conventions used in the raw incoming data (i.e. empty string) are transformed to NULL
  • date are transformed using the ISO8601 in YYYY-MM-DD hh:mm:ss
  • url are formatted as following http[s]://www.yoururl.com
  • normalization vocabularies are used to normalize special cases that will be continuously added to ad-hoc vocabularies, such as accented letters, specific names, etc. The terms in the vocabularies are substituted with the appropriate translation, and this will be done both for all dataset (general normalization vocabularies), or for owner/dataset (source normalization vocabularies and dataset normalization vocabularies).
  • address written in a single cell, is interpreted (when possible) and rearranged as follows: {address type} {address name}, {civic number}, {zip code} {City} ({Provincia/State}), Country. The system recognize it as address, via semantic tag.
• new columns from enrichment are named as following: __{enrichment type}[_{column name}], where text in {} is mandatory, and text in [] is optional and depends on the type of transformation (some are not connected to an existing column)
  • a unique row id (Enrichment) is added via a new column called __ROWID to ensure global uniqueness of the id within DAF. It is made as following: {dataset name space}_{row hash} [TBD]
  • a generated datetime (Enrichment) is added with the timestamp of the ingestion in a column called __dtcreated
  • a updated datetime (Enrichment) is added with the timestamp of the last update for that row in a column called __dtupdate
  • a controlled vocabulary standardization (Standardization) columns is added every time there is a feature/column that is tagged to have a controlled vocabulary associated (the tagging happens in the Catalog Manager, at metadata level). In this case, a procedure is activated to check whether the values contained in the column obey to the vocabulary. This will result in the addition of two columns: __std_{col name} will have the value of the original column in case they are found in the vocabulary, or the closest value (according to some distance metrics like the Levenshtein distance) in case the value is not exactly found; __stdtat_{col name} containing the distance between the original value and the closest one found in the vocabulary. Be aware that the final dataset will not contain the __std_{col name} column, as its value will be put in a column named with the original name {col name}. The __std_{col name} column is used in intermediate steps of the ingestion process.
  • a unique identifier / code (Enrichment) for the standardized columns, in case the controlled vocabulary has a code associated to the standard label. This column will be called __stdcode_{column name}
  • address components (Enrichment): every time there is a column/field tagged as address, this enrichment step will add the following columns inferred from it:
    • __address@placetype_{column name}: it contains the type of address (i.e. via, piazza, viale, ecc.) contained in the address from the column {column name}
    • __address@placename_{column name}: it contains the name of the address (i.e. ‘del Corso’) contained in the address from the column {column name}
    • __address@placecode_{column name}: it contains the code (unique identifier) of the address from its controlled vocabulary.
    • __address@cityname_{column name}: it contains the name of the city of the address from its controlled vocabulary.
    • __address@citycode_{column name}: it contains the code (unique identifier) of the city from its controlled vocabulary.
    • __address@provname_{column name}: it contains the name of the ‘provincia’ or state of the address from its controlled vocabulary.
    • __address@provcode_{column name}: it contains the code (unique identifier) of the ‘provincia’ or state from its controlled vocabulary.
    • __address@countryname_{column name}: it contains the name of the country of the address from its controlled vocabulary.
    • __address@countrycode_{column name}: it contains the code (unique identifier) of the country from its controlled vocabulary.
    • __address@lat_{column name}: it contains the latitude of the address.
    • __address@lon_{column name}: it contains the longitude of the address.

Ingestion pipeline steps¶

The following steps describe the ingestion pipeline applying the above conventions to the incoming new data.

1. New data are captured from the incoming source and stored into a landing area into DAF HDFS.
2. Normalization procedures are applied to the incoming data so to apply DAF conventions. The result will be a dataset with new features/columns added named __norm_{col name}, containing the result of the normalization applied to all features/columns of the original dataset.
3. Standardization procedures are applied to the normalized columns of the previous step. The result will be a dataset generated a the previous step, with the addition of the standardized columns (only when the standardization can be applied, so the number of added columns may be less then the total number of columns in the original dataset (step 1.) named __std_{col name}.
4. Enrichment procedures are applied to the dataset at step 3, resulting in a new dataset build starting from the one generated in step 3 and adding enrichment columns named as follows __{enrichment tuype}[_{col name}].
5. Finalization procedure takes as input the dataset in step 4 and rearrange its columns as follows: it contains the normalized and standardized columns, named as the original columns in the incoming dataset (step 1); the original raw data columns named __raw_{col name}, the list of enrichment columns from step 4.

Final Dataset Structure¶

Based on the conventions and ingestion pipelines described above, a final dataset will have the following structure:

• `ROWID` column, with a unique identifier for the row.
• List of dataset columns: the list of columns of the original raw dataset ingested, at which all the procedures described above have been applied. These columns are named with the original column names provided at the ingestion time.
• List of original raw columns: this part reproduce the original data as provided for the ingestion. These columns will be named as follows: __raw_{col name}.
• List of enrichment columns: those are columns that add additional information to the dataset, extracted at ingestion time. They will be named according to the following convention: __{enrichment type}[_{col name}].

As an example, consider the following dataset to be ingested into DAF.

Step 1: Ingest raw incoming data into DAF, after creation of dataset instance in the catalog manager.

Among other things, let’s suppose that the following metadata has been associated to the dataset:

• industry_type has been associated with the ATECO contolled vocabulary
• address has been linked to the semantic tag associated to address, and to data type ‘address’
• city has been linked to the semantic tag associated to the city and connected controlled vocabulary
• state has been linked to the semantic tag associated to the state and connected controlled vocabulary
• website has been associated to the data type url

Step 2: Apply normalization procedures

Step 3: Apply standardization procedures

Step 4: Apply enrichment procedures

Step 5: Finalization

Data at rest security policies¶

The default storage platform is HDFS, so once the data has been put on HDFS must be protected using the proper permissions. Moreover, since the data need to be accessible through Impala, a set of proper permissions should be provided to Sentry to open the data to the authorized users.

Regardless the data is accessed either from HDFS or through Impala the security policies should be the same. That means that the security rules defined regardless the particular data acces and should be compiled into prpper permissions rule either on HDFS or Sentry.

CREATE EXTERNAL TABLE  domain__subdomain.dataset_stage_format
(
   ...
)
...
LOCATION '/daf/ordinary/sourceOrg/domain/subdomain/dataset.stage.format';

CREATE ROLE db_domain__subdomain_role;
GRANT ALL ON DATABASE domain__subdomain TO ROLE db_domain__subdomain_role;
GRANT ROLE db_domain__subdomain_role TO GROUP dataowner;
INVALIDATE METADATA;

CREATE ROLE table_domain__subdomain__dataset_stage_format_role;
GRANT SELECT ON TABLE domain__subdomain.dataset_stage_format TO ROLE table_domain__subdomain__dataset_stage_format_role;
GRANT ROLE table_domain__subdomain__dataset_stage_format_role TO GROUP dataowner;
INVALIDATE METADATA;

Virtual Machine Account¶

USER:	user
PASSWORD:	password

Check the IP address assigned to the Virtual Machine (also check bridge of virtual machines before start if you use wireless or Ethernet adapter on your PC) using the command:

> ip a | grep enp0s3

Access the Virtual Machine via ssh:

> ssh user@xxx.xxx.xxx.xxx (Virtual Machine IP)

Docker image¶

In the Virtual Machine, to access a folder with container docker image:

> sudo -i
> cd /root/docker

All the containers start automatically when the Virtual Machine starts.

Configuration¶

In your PC Hosts file, add the following lines:

x.x.x.x ipa.example.test superset.daf.test.it metabase.daf.test.it ckan.daf.test.it mongodb ckan metabase supersetd
127.0.0.1 datipubblici-private.daf.test.it

where x.x.x.x is the Virtual Machine IP address.

Services¶

In the host, run the following command to clone the DAF project:

> git clone https://github.com/italia/daf.git

In case sbt is not found, install it:

> echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list
> sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823
> sudo apt-get update
> sudo apt-getinstall sbt

> sbt
> clean
> compile
> publishLocal

> sbt
> clean
> compile
> run -Dconfig.resource=svil.conf -Dhttp.port=9002

> sbt
> clean
> compile
> run -Dconfig.resource=svil.conf -Dhttp.port=9001

> git clone  https://github.com/italia/daf-dataportal-backend

> sbt
> clean
> compile
> run -Dconfig.resource=local.conf

> git clone  https://github.com/italia/daf-dataportal

apiURLSSOManager: "http://localhost:9002/sso-manager",
apiURLDatiGov: "http://localhost:9000/dati-gov/v1",
apiURLCatalog: "http://localhost:9001/catalog-manager/v1",
apiURLIngestion: "http://localhost:9002/ingestion-manager/v1",
apiURLSecurity: "http://localhost:9002/security-manager/v1",
urlMetabase: 'http://metabase.daf.test.it',
urlSuperset: 'http://superset.daf.test.it',

domain:".daf.test.it"

"start": "PORT=80 react-scripts start"

npm install
npm start

npm run build
npm install -g serve
serve -s build

Local Installation¶

• Pre-requisites: JDK 8, SBT, GIT client
• git clone https://github.com/italia/daf
• cd daf/common
• sbt publishLocal
• cd ../catalog_manager
• sbt
• run
• connect to http://localhost:9000/catalog-manager

You should see a swagger UI with all endpoints described. Nevertheless, the authorization is not required by the UI you should pass at least a Basic authorization token made by an equal user name and password.

To test the endpoints, we suggest to use a tool like Postman

DAF integration note¶

[TBD]

Endpoints¶

[TBD]

Local Installation¶

• Pre-requisites: JDK 8, SBT, GIT client
• git clone https://github.com/italia/daf
• cd daf/common
• sbt publishLocal
• cd ../ingestion_manager
• sbt
• run
• connect to http://localhost:9000/ingestion-manager

You should see a swagger UI with all endpoints described. Nevertheless, the authorization is not required by the UI you should pass at least a Basic authorization token made by an equal user name and password.

To test the endpoints we suggest to use a tool like Postman

DAF integration note¶

[TBD]

Endpoints¶

[TBD]

Local Installation¶

• Pre-requisites: JDK 8, SBT, GIT client
• git clone https://github.com/italia/daf
• cd daf/common
• sbt publishLocal
• cd ../security_manager
• sbt
• run
• connect to http://localhost:9000/security-manager

You should see a swagger UI with all endpoints described. Nevertheless, the authorization is not required by the UI you should pass at least a Basic authorization token made by an equal user name and password.

To test the endpoints we suggest to use a tool like Postman

DAF integration note¶

[TBD]

Endpoints¶

[TBD]

Local Installation¶

• Pre-requisites: JDK 8, SBT, GIT client
• git clone https://github.com/italia/daf
• cd daf/common
• sbt publishLocal
• cd ../dataset_manager
• sbt
• run
• connect to http://localhost:9000/dataset-manager

You should see a swagger UI with all endpoints described. Nevertheless, the authorization is not required by the UI you should pass at least a Basic authorization token made by an equal user name and password.

To test the endpoints we suggest to use a tool like Postman

DAF integration note¶

[TBD]

Endpoints¶

[TBD]

Local installation¶

• git clone https://github.com/italia/daf
• cd daf/common
• sbt publishLocal
• cd ../daf/storage_manager
• sbt
• run
• connect to http://localhost:9000/storage-manager

DAF integration note¶

[TBD]

Endpoints¶

[TBD]

What is it?¶

The Frontend Manager is the layer that serves functionalities to the frontend web applications (Dataportal-public and Dataportal-private). It integrates external applications (e.g. CKAN) and exposes REST API to the web apps.

It has been developed using the following technologies:

• playframework 2.5.12
• scala 2.11.8
• sbt 0.13
• Zalando’s api-first-hand

Install¶

• $ git clone git@github.com:italia/dati-frontendserver.git
• $ sbt compile
• $ sbt run
• Connect to http://localhost:9000

Setup¶

…

What is it?¶

The Dataportal-public is the web app that allows access to the open data catalog and other content that can be exposed publicly.

Install¶

Before proceeding with the installation steps, you need to install and run the following external components:

> $ git clone https://github.com/italia/daf-dataportal-backend

> npm install
> npm start

> npm run mock

> npm run build
> npm install -g serve
> serve -s build

What is it?¶

The Dataportal-private is the web app that allows access to the functionalities of DAF, like:

• Ingestion form to add dataset with metadata
• Business Intelligence with Superset (AirBnB)
• Graphs with Metabase
• Data science with Jupyter + Sparkmagic
• Ontologies and Controlled Vocabularies repository

Install¶

Before proceeding with the installation steps, you need to install and run the following external components:

> git clone https://github.com/italia/daf-dataportal

> npm install
> npm start

> npm run mock

> npm run build
> npm install -g serve
> serve -s build

Docker¶

FreeIPA server can be run in a Docker container for testing or demo purposes. It makes it possible to run all the processes comprising the server in an isolated way, leaving the host free to run other software, not clashing with the FreeIPA server.

This installation is done on Ubuntu 16.04. FreeIPA is focused on Linux (and other standards compliant) systems. Therefore, in our knowledge, you cannot run a container of a FreeIPA server on Mac OS or Windows. However, any help in this direction is very welcomed!!

Follow these steps to run our FreeIPA server docker:

1. Create a directory which will hold the server data:

> mkdir /var/lib/ipa-data

2. Edit /etc/hosts and ensure that the IPA server address is listed. This is required for Apache to work properly. You have to change IPA_SERVER_IP with the IPA server IP:

127.0.0.1      localhost localhost.localdomain localhost4 localhost4.localdomain4
::1            localhost localhost.localdomain localhost6 localhost6.localdomain6
IPA_SERVER_IP    ipa.example.test

3. Finally, You run the container:

> docker run -it -p 389:389 -p 443:443 -p 636:636 --name freeipaldap --cap-add SYS_ADMIN --security-opt seccomp:unconfined -v /sys/fs/cgroup:/sys/fs/cgroup:ro --tmpfs /run --tmpfs /tmp -v /var/lib/ipa-data:/data:Z -h ipa.example.test italia/freeipa-server --ds-password=The-directory-server-password --admin-password=The-admin-password

where:

• –cap-add SYS_ADMIN, performs a range of system administration operations (see here for more details ).
• –security-opt seccomp:unconfined to run a container without the default seccomp profile (see here for more details ).
• -v /var/lib/ipa-data:/data:Z to store data and configurations in the folder /var/lib/ipa-data/

Answer to the question:

Do you want to configure integrated DNS (BIND)? [no]: –> press “Enter”

Server host name [ipa.example.test]: –> press “Enter”

Please confirm the domain name [example.test]: –> press “Enter”

Please provide a realm name [EXAMPLE.TEST]: –> press “Enter”

Continue to configure the system with these values? [no]: –> type “y” and press “Enter”

Wait some time until FreeIPA server is completely configured and started. The server is ready when on the shell the following message appears:

> FreeIPA server configured.

• You can connect to FreeIPA Server from a web interface:

  https://IPA_SERVER_IP:443

  USER: admin

  PW: adminpassword

• You can also connect with an LDAP client with Server IP address IPA_SERVER_IP

• The container can then be started and stopped with the following commands:

> docker stop freeipaldap
> docker start freeipaldap

References¶

[1] FreeIpa docker-hub documentation.

[2] Using Free Ipa for user authentication.

[3] FreeIpa website.

FreeIpa Instance¶

We installed a FreeIpa server which can be used for test purposes. It can be reached at the address 91.206.129.245.

Account Management Dependency¶

This configuration of CKAN needs an account management system to work with. We provide three different options, you will find more info on their respective sections:

• Local LDAP Docker
• Local FreeIPA Docker (works only with Linux)
• Remote FreeIpa Server

Ckan docker compose¶

Now that we have a LDAP server up we can run the CKAN Docker Compose. It will run an instance of Solr, Postgresql, Redis and Mongo.

First we have to build a custom image:

> cd ./daf-recipes/ckan
> ./build_local.sh

Then edit the file ckan.ini:

• If you are using our openLDAP server:

  # LDAP Intergration with ldap and ip address
  ckanext.ldap.uri = ldap://LDAP_IP:389
  ckanext.ldap.auth.dn = cn=admin,dc=daf,dc=test,dc=it
  ckanext.ldap.auth.password = admin
  ckanext.ldap.base_dn = cn=users,cn=accounts,dc=daf,dc=test,dc=it
  ckanext.ldap.search.filter = uid={login}
  ckanext.ldap.username = uid
  ckanext.ldap.email = mail
  ckanext.ldap.ckan_fallback = True
  

where LDAP_IP is the IP of the LDAP docker. To know the LDAP IP, run:

> docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' ldap
172.22.0.2

We know that this is not the best approach to connect containers among them (maybe it is the worst), we are using a deprecated compose file version (i.e. version 1 rather than using version 3), and we are using very heavy images. We will improve CKAN docker compose as soon as possible.

• If you are using our FreeIpa server

  # LDAP Intergration with ldap and ip address
  ckanext.ldap.uri = ldap://91.206.129.245:389
  ckanext.ldap.auth.dn = uid=admin,cn=users,cn=accounts,dc=daf,dc=test,dc=it
  ckanext.ldap.auth.password = aiyaiPh8
  ckanext.ldap.base_dn = cn=users,cn=accounts,dc=daf,dc=test,dc=it
  ckanext.ldap.search.filter = uid={login}
  ckanext.ldap.username = uid
  ckanext.ldap.email = mail
  ckanext.ldap.ckan_fallback = True
  

Now that CKAN container is up, type http://localhost:5000 on your browser and login as user bob (password password).

Account Management Dependency¶

This configuration of CKAN needs an account management system to work with. We provide three different options, you will find more info on their respective sections:

• Local LDAP Docker
• Local FreeIPA Docker (works only with Linux)
• Remote FreeIpa Server

JupyterHub¶

This Docker container runs a JupyterHub instance which is connected with a PostgreSQL database.

Run the Docker container:

> cd ./daf-recipes/jupyterhub
> docker-compose up -d

Check whether dockers are running:

> docker ps
8350963ac06c        jupyterhub_jupyterhub   "/wait_db_is_ready.sh"   16 minutes ago      Up 16 minutes       0.0.0.0:8000->8000/tcp                       jupyterhub
6a0d0d6c3b9a        osixia/phpldapadmin     "/container/tool/run"    17 minutes ago      Up 17 minutes       0.0.0.0:80->80/tcp, 443/tcp                  phpldapadmin
e8ff9611aeff        osixia/openldap         "/container/tool/r..."   17 minutes ago      Up 17 minutes       0.0.0.0:389->389/tcp, 0.0.0.0:636->636/tcp   ldap
cee2d35feaaf        postgres:9.6            "docker-entrypoint..."   2 hours ago         Up 2 hours          0.0.0.0:5432->5432/tcp                       postgresjupyterhub

To open the interactive shell type http://localhost:8000 and login as user alice (password password).