ADR on trino catalogs

modules/contributor/pages/adr/ADR019-trino_catalog_definitions.adoc

@@ -0,0 +1,122 @@
+= ADR019: Trino catalog definitions
+Sebastian Bernauer <sebastian.bernauer@stackable.de>
+v0.1, 16.05.2022
+:status: accepted
+
+* Status: {status}
+* Deciders:
+** Felix Hennig
+** Malte Sander
+** Sebastian Bernauer
+** Sönke Liebau
+** Teo Klestrup Röijezon
+* Date: 17.05.2022
+
+== Context and Problem Statement
+
+https://trino.io[Trino] allows user to specify multiple catalogs to connect to a variety of different data-sources.
+We need to agree on a mechanism to
+
+1. Specifying Trino catalog definitions (this ADR)
+2. Connect a catalog definition to an Trino cluster (xref:adr/ADR020-trino_catalog_usage.adoc[])
+
+== Decision Drivers
+
+* Multiple different types of connectors must be supported, e.g. `Hive`, `Iceberg`, `Oracle` and `PostgreSQL`.
+* In case of catalogs that use distributed file-systems such as HDFS or S3 the access needs to be configured.
+
+== Considered Options
+
+* TrinoCatalog CRD with discovery ConfigMaps from same namespace
+* TrinoCatalog CRD with discovery ConfigMaps from potentially other namespaces
+
+== Decision Outcome
+
+Chosen option: "TrinoCatalog CRD with discovery ConfigMaps from same namespace".
+This option lines up with the way we want to handle discovery ConfigMaps:
+The operator of the service connecting to (e.g. hdfs) watches for HdfsDirectory objects and provides us with a discovery ConfigMap in the target namespace.
+We will start only implementing the `Hive` connector and support more connectors in the future.
+
+== Pros and Cons of the Options
+A TrinoCatalog has a top-level complex enum to distinguish between the different connector types.
+This way every connector can define it's own set of attributes that it supports.
+
+=== TrinoCatalog CRD with discovery ConfigMaps from same namespace
+Here all references to discovery ConfigMaps such as HDFS or Hive only are a string that contains the name of the ConfigMap. The ConfigMap must reside in the same namespace as the TrinoCatalog object
+
+[source,yaml]
+----
+---
+# Pseudo code!
+TrinoCatalog
+metadata:
+    name: trino-catalog
+    namespace: default
+spec:
+    hive:
+        metastore: # mandatory
+            configMap: my-hive-metastore
+        s3: # S3ConnectionDef, optional
+            inline:
+                host: minio
+            # OR
+            reference: my-minio-connection
+        hdfs: # optional
+            configMap: my-hdfs # will provide hdfs-site.xml
+            impersonation: true # optional, defaults to false
+            # there is no kerberos or wireEncryption attribute, as the information about kerberos comes from the discovery configmap
+    # OR
+    iceberg: {} # Attributes need to be defined later on when we support iceberg
+    # OR
+    postgresql: {} # Attributes need to be defined later on when we support postgresql
+    # OR [...]
+----
+
+Looking at the example of hdfs the hdfs discovery ConfigMap will be created by the hdfs-operator.
+That can be the case because we are running in the same namespace as hdfs or we place a HdfsDirectory object into the Trino namespace.
+The hdfs-operator then detects the HdfsDirectory object and places a discovery ConfigMap into the Trino namespace.
+(This is similar to the way ZooKeeper's ZNodes currently work)
+
+* Good, because it's simple and we don't have to worry about cross-namespace access
+* Bad, because it prohibits the usage of an HDFS in a different namespace than the TrinoCatalog namespace. This can be solved by letting the hdfs-operator put the same discovery configmap into multiple namespaces (including the one with the TrinoCatalog)
+
+=== TrinoCatalog CRD with discovery ConfigMaps from potentially other namespaces
+Here all references to discovery ConfigMaps such as HDFS or Hive are a tuple of the name and the namespace of the ConfigMap. The namespace is optional, if not provided the same namespace from the TrinoCatalog will be used. The ConfigMap can reside in a different namespace as the TrinoCatalog object
+
+[source,yaml]
+----
+---
+# Pseudo code!
+TrinoCatalog
+metadata:
+    name: trino-catalog
+    namespace: default
+spec:
+    hive:
+        metastore: # mandatory
+            configMap:
+                name: my-hive-metastore
+                namespace: default # optional
+        s3: # S3ConnectionDef, optional
+            inline:
+                host: minio
+            # OR
+            reference:
+                name: my-minio-connection
+                namespace: default # optional
+        hdfs: # optional
+            configMap: # will provide hdfs-site.xml
+                name: my-hdfs
+                namespace: default # optional
+            impersonation: true # optional, defaults to false
+            # there is no kerberos or wireEncryption attribute, as the information about kerberos comes from the discovery configmap
+    # OR
+    iceberg: {} # Attributes need to be defined later on when we support iceberg
+    # OR
+    postgresql: {} # Attributes need to be defined later on when we support postgresql
+    # OR [...]
+----
+
+* Good, because it allows easy cross-namespace access
+* Bad, because it's more complicated
+* Bad, because we can't simply mount the ConfigMap (e.g. with hdfs-site.xml) but instead somehow need to "transfer" it between different namespaces and watch the original ConfigMap.

modules/contributor/pages/adr/ADR020-trino_catalog_usage.adoc

@@ -0,0 +1,80 @@
+= ADR020: Trino catalog usage
+Sebastian Bernauer <sebastian.bernauer@stackable.de>
+v0.1, 16.05.2022
+:status: accepted
+
+* Status: {status}
+* Deciders:
+** Felix Hennig
+** Malte Sander
+** Sebastian Bernauer
+** Sönke Liebau
+** Teo Klestrup Röijezon
+* Date: 17.05.2022
+
+== Context and Problem Statement
+
+https://trino.io[Trino] allows user to specify multiple catalogs to connect to a variety of different data-sources.
+We need to agree on a mechanism to
+
+1. Specifying Trino catalog definitions (xref:adr/ADR019-trino_catalog_definitions.adoc[])
+2. Connect a catalog definition to an Trino cluster (this ADR)
+
+== Decision Drivers
+
+* Catalogs must somehow be added to Trino clusters
+* Catalogs should be reusable between multiple Trino clusters - e.g. users have two identical Trino clusters - one for adhoc queries, on for scheduled jobs.
+
+== Considered Options
+
+* Catalog references Clusters
+* Cluster references Catalogs
+* Mapping object between Catalog and Cluster
+* Catalog references Cluster
+* Mapping via labels and label selectors
+
+== Decision Outcome
+
+Chosen option: "Mapping via labels and label selectors", because it's the most flexible solution and delegates implementation details to kubernetes.
+
+== Pros and Cons of the Options
+=== Catalog references Clusters
+image::adr/20_option1.png[]
+
+* Good, because if Trino instances in different stages have different catalogs, a Trino Object can be reused over the different stages
+* Bad, because if a Trino Cluster has multiple catalogs and you want a similar Trino Cluster you need to modify all catalogs (and maybe restart your first Trino multiple times), not just simply create a new Trino Cluster copy
+
+=== Cluster references Catalogs
+image::adr/20_option2.png[]
+
+* Good, because it's the normal flow that our Product CRDs point to other objects, not the other direction
+* Bad, because people adding new catalogs need to be able to modify the `TrinoCluster` object. There may be companies out there where different people operate Trino and manage the catalogs
+* Bad, because if Trino instances in different stages have different catalogs a Trino Object can not be reused over the different stages
+
+=== Mapping object between Catalog and Cluster
+image::adr/20_option3.png[]
+
+* Bad, because more complicated for the users
+* Bad, because more complicated watches needed
+
+=== Catalog references Cluster
+This is the same as <<_catalog_references_clusters>> but instead of a list of Clusters the Catalog only contains a single Cluster.
+
+image::adr/20_option4.png[]
+
+* Good, because if Trino instances in different stages have different catalogs a Trino Object can be reused over the different stages
+* Good compared to <<_catalog_references_clusters>>, because an `TrinoCatalog` is associated with an `TrinoCluster` and the cluster can add additional information to it like the current deployment status
+* Bad, because catalogs can not be reused between multiple Trino clusters
+
+
+=== Mapping via labels and label selectors
+This is option <<_mapping_object_between_catalog_and_cluster>> the "kubernetes way".
+
+Every `TrinoCatalog` object provides a set of labels.
+The `TrinoCluster` objects provide a LabelSelector which defines which catalogs should be included in the Trino instance.
+
+image::adr/20_option5.png[]
+
+* Good, because flexible (Trino cluster administrators can add new catalogs and teams can add their needed catalogs to an managed Trino instance)
+* Good, because the usage of well-known kubernetes patterns
+* Good, because delegates implementation details to kubernetes (e.g. easier watches)

modules/contributor/partials/current_adrs.adoc

@@ -15,3 +15,5 @@
 **** xref:adr/ADR016-definition_of_s3_objects.adoc[]
 **** xref:adr/ADR017-tls_authentication.adoc[]
 **** xref:adr/ADR018-product_image_versioning.adoc[]
+**** xref:adr/ADR019-trino_catalog_definitions.adoc[]
+**** xref:adr/ADR020-trino_catalog_usage.adoc[]