Catalog Providers

Catalog Providers

Note

Provides different ways to specify DataSet Properties to Read/Write Data API
Can be set using gimel.catalog.provider property
Can be set to USER/HIVE/UDC

Catalog Provider as USER

Note

Uses DataSetProperties specified by user as Json String or DataSetProperties Object
Recommended to be used only for testing/exploring purposes

GSQL

Set the catalog provider property to USER

%%sql 
set gimel.catalog.provider=USER

Set DataSetProperties Json

Note: The property name should be {datasetName}.dataSetProperties. For Example if your dataset name is pcatalog.mydataset, then the property name would be pcatalog.mydataset.dataSetProperties.

Example for HBase Dataset

%%sql
set pcatalog.hbase_test.dataSetProperties={
    "datasetType": "HBASE",
    "fields": [
        {
            "fieldName": "id",
            "fieldType": "string",
            "isFieldNullable": false
        },
        {
            "fieldName": "name",
            "fieldType": "string",
            "isFieldNullable": false
        },
        {
            "fieldName": "address",
            "fieldType": "string",
            "isFieldNullable": false
        },
        {
            "fieldName": "age",
            "fieldType": "string",
            "isFieldNullable": false
        },
        {
            "fieldName": "company",
            "fieldType": "string",
            "isFieldNullable": false
        },
        {
            "fieldName": "designation",
            "fieldType": "string",
            "isFieldNullable": false
        },
        {
            "fieldName": "salary",
            "fieldType": "string",
            "isFieldNullable": true
        }
    ],
    "partitionFields": [],
    "props": {
        "gimel.hbase.rowkey":"id",
        "gimel.hbase.table.name":"adp_bdpe:test_annai",
        "gimel.hbase.namespace.name":"namespace",
        "gimel.hbase.columns.mapping":"personal:name,personal:address,personal:age,professional:company,professional:designation,professional:salary"
    }
}

Run the sql

%%pcatalog
select * from pcatalog.hbase_test

DATA API Usage

Imports and Initializations

import org.apache.spark.sql.{DataFrame, SQLContext};
import org.apache.spark.sql.hive.HiveContext;
import org.apache.spark.{SparkConf, SparkContext};
import org.apache.spark.rdd.RDD;
import com.paypal.gimel.logger.Logger;
import com.paypal.gimel.{DataSet, DataSetType};
import spray.json.DefaultJsonProtocol._;
import spray.json._;
val logger = Logger(this.getClass.getName)
import org.apache.spark.sql.{DataFrame, SaveMode, SQLContext}
import com.paypal.gimel.common.catalog._

//Initiate HiveContext
val hiveContext = new HiveContext(sc);
val dataSet = com.paypal.gimel.DataSet(spark);
val sqlContext = hiveContext.asInstanceOf[SQLContext]

Usage

hiveContext.sql("set gimel.catalog.provider=USER");

val datasetPropsJson="""
{
    "datasetType": "ELASTIC_SEARCH",
    "fields": [],
    "partitionFields": [],
    "props": {
        "gimel.es.index.partition.isEnabled": "true",
        "gimel.es.index.partition.suffix": "20180120",
        "datasetName": "data_set_name",
        "nameSpace": "pcatalog",
        "es.port": "8080",
        "es.resource": "index/type",
        "gimel.es.index.partition.delimiter": "-",
        "es.nodes": "http://es_host"
    }
}
"""

val data = dataSet.read("pcatalog.data_set_name",
                         Map("pcatalog.data_set_name.dataSetProperties" -> datasetPropsJson))

Catalog Provider as HIVE

Note

Uses Dataset Properties from the external hive table
Set the catalog provider property to HIVE

%%sql 
set gimel.catalog.provider=HIVE

Create Hive Table pointing to physical storage

You can find hive table templates for each storage in their docs Stack & Version Compatibility

Catalog Provider as UDC (Unified Data Catalog)

Note

Uses DataSetProperties from the UDC
In order to use this, storage system of the Dataset should be onboarded in UDC
Set the catalog provider property to UDC

%%sql 
set gimel.catalog.provider=UDC

UDC parameters

Property	Mandatory?	Description	Example	Default
rest.service.method or spark.rest.service.method	Y	UDC Rest service protocol	http/https	https
rest.service.host or spark.rest.service.host	Y	UDC Rest service host name	my-udc.example.com	None
rest.service.port or spark.rest.service.port	Y	UDC Rest service port	80	443
gimel.udc.auth.enabled or spark.gimel.udc.auth.enabled	N	UDC auth enabled flag	true/false	true
gimel.udc.auth.provider.class or spark.gimel.udc.auth.provider.class	N	This is a Must if `gimel.udc.auth.enabled=true`	com.example.MyAuthProvider	None

UDC Auth

In order to connect to UDC for getting DatasetProperties, UDC-APP-NAME and UDC-AUTH-KEY has to be set in header for GET calls.
Add a custom AuthProvider in order to retrieve the Auth token (UDC-AUTH-KEY value) from any service or key store.
Add gimel-common dependency in your project

  <dependency>
    <groupId>com.paypal.gimel</groupId>
    <artifactId>gimel-common</artifactId>
    <version>${gimel.version}</version>
   </dependency>

Extend com.paypal.gimel.common.security.AuthProvider in gimel-common and implement getCredentials method which should return the auth token. Example:

package com.example

import com.paypal.gimel.common.security._

class MyAuthProvider extends AuthProvider {

  override def getCredentials(props: Map[String, Any]): String = {
    /*
     * Implement your own logic here
     */
  }
}

Here props are the gimel properties passed at runtime.

import org.apache.spark.sql.{Column, Row, SparkSession,DataFrame}
import org.apache.spark.sql.functions._

// Create Gimel SQL reference
val gsql: (String) => DataFrame = com.paypal.gimel.sql.GimelQueryProcessor.executeBatch(_: String, spark)
gsql("set gimel.logging.level=CONSOLE")

val options = Map("rest.service.method" -> "https",
"rest.service.host" -> "my-udc.example.com",
"rest.service.port" -> "443",
"gimel.deserializer.class" -> "com.paypal.gimel.deserializers.generic.JsonDynamicDeserializer",
"gimel.kafka.throttle.batch.fetchRowsOnFirstRun" -> "1",
"gimel.udc.auth.provider.class" -> "com.example.MyAuthProvider")

val df = dataset.read("udc.Kafka.Gimel_Dev.default.flights", options)
df.show