diff --git a/docs/examples/milvus/quickstart/distributed.yaml b/docs/examples/milvus/quickstart/distributed.yaml new file mode 100644 index 000000000..f0f9aa17a --- /dev/null +++ b/docs/examples/milvus/quickstart/distributed.yaml @@ -0,0 +1,23 @@ +apiVersion: kubedb.com/v1alpha2 +kind: Milvus +metadata: + name: milvus-cluster + namespace: demo +spec: + version: "2.6.11" + objectStorage: + configSecret: + name: my-release-minio + topology: + mode: Distributed + distributed: + mixcoord: + replicas: 2 + storageType: Durable + storage: + accessModes: + - ReadWriteOnce + storageClassName: local-path + resources: + requests: + storage: 10Gi diff --git a/docs/guides/milvus/README.md b/docs/guides/milvus/README.md new file mode 100644 index 000000000..ef8652c30 --- /dev/null +++ b/docs/guides/milvus/README.md @@ -0,0 +1,71 @@ +--- +title: Milvus +menu: + docs_{{ .version }}: + identifier: milvus-readme + name: Milvus + parent: milvus-guides + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +url: /docs/{{ .version }}/guides/milvus/ +aliases: + - /docs/{{ .version }}/guides/milvus/README/ +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Overview + +KubeDB supports vector database deployment with Milvus using the `Milvus` CRD. + +## Supported Milvus Features + +| Features | Availability | +|----------------------------------|:------------:| +| Standalone provisioning | ✓ | +| Distributed provisioning | ✓ | +| Monitoring | ✓ | +| TLS | No | +| Ops Requests | No | + +## Example Milvus Manifest + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: Milvus +metadata: + name: milvus-cluster +spec: + version: "2.6.11" + objectStorage: + configSecret: + name: "my-release-minio" + topology: + mode: Distributed + distributed: + mixcoord: + replicas: 2 + storageType: Durable + storage: + accessModes: + - ReadWriteOnce + storageClassName: local-path + resources: + requests: + storage: 10Gi +``` + +## User Guide + +- [Quickstart Milvus](/docs/guides/milvus/quickstart/quickstart.md) with KubeDB operator. +- [Milvus CRD Concept](/docs/guides/milvus/concepts/milvus.md). +- [MilvusVersion CRD Concept](/docs/guides/milvus/concepts/catalog.md). +- [MilvusOpsRequest CRD Concept](/docs/guides/milvus/concepts/opsrequest.md). +- [RBAC Quickstart](/docs/guides/milvus/quickstart/rbac.md) +- [Private Registry](/docs/guides/milvus/private-registry/using-private-registry.md) +- [Custom RBAC](/docs/guides/milvus/custom-rbac/using-custom-rbac.md) +- [Custom Configuration](/docs/guides/milvus/configuration/using-config-file.md) +- [Monitoring](/docs/guides/milvus/monitoring/overview.md) for metrics collection guidance. +- [Builtin Prometheus Monitoring](/docs/guides/milvus/monitoring/using-builtin-prometheus.md) +- [Prometheus Operator Monitoring](/docs/guides/milvus/monitoring/using-prometheus-operator.md) \ No newline at end of file diff --git a/docs/guides/milvus/_index.md b/docs/guides/milvus/_index.md new file mode 100644 index 000000000..2b29e986f --- /dev/null +++ b/docs/guides/milvus/_index.md @@ -0,0 +1,10 @@ +--- +title: Milvus +menu: + docs_{{ .version }}: + identifier: milvus-guides + name: Milvus + parent: guides + weight: 15 +menu_name: docs_{{ .version }} +--- \ No newline at end of file diff --git a/docs/guides/milvus/concepts/_index.md b/docs/guides/milvus/concepts/_index.md new file mode 100644 index 000000000..f522969c1 --- /dev/null +++ b/docs/guides/milvus/concepts/_index.md @@ -0,0 +1,10 @@ +--- +title: Milvus Concepts +menu: + docs_{{ .version }}: + identifier: milvus-concepts + name: Concepts + parent: milvus-guides + weight: 15 +menu_name: docs_{{ .version }} +--- \ No newline at end of file diff --git a/docs/guides/milvus/concepts/catalog.md b/docs/guides/milvus/concepts/catalog.md new file mode 100644 index 000000000..3fabc23ab --- /dev/null +++ b/docs/guides/milvus/concepts/catalog.md @@ -0,0 +1,70 @@ +--- +title: MilvusVersion CRD +menu: + docs_{{ .version }}: + identifier: milvus-catalog-concepts + name: MilvusVersion + parent: milvus-concepts-milvus + weight: 15 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# MilvusVersion + +## What is MilvusVersion + +`MilvusVersion` is the catalog CRD that defines the Milvus engine image and related metadata for KubeDB-managed Milvus deployments. + +KubeDB uses this CRD when resolving `Milvus.spec.version`. + +## MilvusVersion Specification + +```yaml +apiVersion: catalog.kubedb.com/v1alpha1 +kind: MilvusVersion +metadata: + name: 2.6.11 +spec: + db: + image: ghcr.io/appscode-images/milvus:2.6.11 + etcdVersion: v3.5.21 + securityContext: + runAsUser: 1000 + version: 2.6.11 +``` + +### metadata.name + +`metadata.name` is a required field that specifies the name of the `MilvusVersion` CR. You have to specify this name in `spec.version` field of [Milvus](/docs/guides/milvus/concepts/milvus.md) CR. + +We follow this convention for naming MilvusVersion CR: + +- Name format: `{Original Milvus image version}-{modification tag}` + +We use official Apache Milvus release tar files to build docker images for supporting Milvus versions and re-tag the image with v1, v2 etc. modification tag when there's any. An image with higher modification tag will have more features than the images with lower modification tag. Hence, it is recommended to use MilvusVersion CR with the highest modification tag to enjoy the latest features. + +### spec.version + +`spec.version` is a required field that specifies the original version of Milvus database that has been used to build the docker image specified in `spec.db.image` field. + +### spec.db.image + +`spec.db.image` is a required field that specifies the docker image which will be used to create PetSet by KubeDB operator to create expected Milvus database. + +### spec.etcdVersion + +`spec.etcdVersion` specifies the compatible Etcd version required by this Milvus release. + +### spec.deprecated + +`spec.deprecated` is an optional field that specifies whether the docker images specified here is supported by the current KubeDB operator. + +The default value of this field is `false`. If `spec.deprecated` is set to `true`, KubeDB operator will skip processing this CRD object and will add a event to the CRD object specifying that the DB version is deprecated. + +## Next Steps + +- Read the [Milvus CRD concept](/docs/guides/milvus/concepts/milvus.md). +- Run the [Milvus quickstart](/docs/guides/milvus/quickstart/quickstart.md). \ No newline at end of file diff --git a/docs/guides/milvus/concepts/milvus.md b/docs/guides/milvus/concepts/milvus.md new file mode 100644 index 000000000..06bc2fe8a --- /dev/null +++ b/docs/guides/milvus/concepts/milvus.md @@ -0,0 +1,548 @@ +--- +title: Milvus CRD +menu: + docs_{{ .version }}: + identifier: milvus-concepts-milvus + name: Milvus + parent: milvus-concepts + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Milvus + +## What is Milvus + +`Milvus` is a KubeDB `CustomResourceDefinition` used to deploy and manage Milvus vector databases. You only need to describe the desired database configuration in a `Milvus`object, and the KubeDB operator will create Kubernetes objects in the desired state for you. + +## Milvus Spec + +As with all other Kubernetes objects, a Milvus needs `apiVersion`, `kind`, and `metadata` fields. It also needs a `.spec` section. Below is an example of Milvus object. + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: Milvus +metadata: + name: milvus-cluster + namespace: kubedb +spec: + version: "2.6.11" + objectStorage: + configSecret: + name: "my-release-minio" + metaStorage: + externallyManaged: true + endpoints: + - http://etcdcluster-sample-0.etcdcluster-sample.default.svc.cluster.local:2379 + - http://etcdcluster-sample-1.etcdcluster-sample.default.svc.cluster.local:2379 + - http://etcdcluster-sample-2.etcdcluster-sample.default.svc.cluster.local:2379 + disableSecurity: false + authSecret: + name: "milvus-auth" + externallyManaged: true + configuration: + secretName: my-release-user-config + inline: + milvus.yaml: | + log: + level: info + file: + maxAge: 30 + topology: + mode: Distributed + distributed: + mixcoord: + replicas: 2 + podTemplate: + spec: + containers: + - name: milvus + resources: + requests: + cpu: 500m + memory: 1Gi + limits: + cpu: 600m + memory: 2Gi + + datanode: + replicas: 2 + podTemplate: + spec: + containers: + - name: milvus + resources: + requests: + cpu: 600m + memory: 1Gi + limits: + cpu: 700m + memory: 3Gi + + proxy: + replicas: 2 + podTemplate: + spec: + containers: + - name: milvus + resources: + requests: + cpu: 500m + memory: 2Gi + limits: + cpu: 600m + memory: 4Gi + querynode: + replicas: 2 + podTemplate: + spec: + containers: + - name: milvus + resources: + requests: + cpu: 800m + memory: 3Gi + limits: + cpu: 900m + memory: 4Gi + streamingnode: + replicas: 3 + podTemplate: + spec: + containers: + - name: milvus + resources: + requests: + cpu: 600m + memory: 2Gi + limits: + cpu: 700m + memory: 2Gi + storageType: Durable + storage: + accessModes: + - ReadWriteOnce + storageClassName: local-path + resources: + requests: + storage: 2Gi + monitor: + agent: prometheus.io/operator + prometheus: + exporter: + port: 9091 + resources: + limits: + memory: 512Mi + requests: + cpu: 600m + memory: 256Mi + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + runAsGroup: 1000 + runAsNonRoot: true + runAsUser: 1000 + seccompProfile: + type: RuntimeDefault + serviceMonitor: + interval: 10s + labels: + release: prometheus + tls: + issuerRef: + name: milvus-issuer + kind: Issuer + apiGroup: "cert-manager.io" + external: + mode: mTLS + internal: + mode: TLS + deletionPolicy: WipeOut + healthChecker: + periodSeconds: 15 + timeoutSeconds: 10 + failureThreshold: 2 + disableWriteCheck: false +``` + +### spec.version + +`spec.version` is a required field specifying the name of the [MilvusVersion](/docs/guides/milvus/concepts/milvusversion.md) crd where the docker images are specified. Currently, when you install KubeDB, it creates the following `Milvus` resources, + +- `2.6.7` +- `2.6.9` +- `2.6.11` + +### spec.objectStorage + +`spec.objectStorage` is a required field that specifies the object storage backend used by Milvus. Milvus depends on external object storage (such as MinIO or any S3-compatible service) to store its data. + +The configuration is provided via a Kubernetes `Secret` referenced in this field: + +```yaml +objectStorage: + configSecret: + name: my-release-minio +``` +The referenced secret must contain the following keys: + +- address – endpoint of the object storage service +- accesskey – username for authentication +- secretkey – password for authentication + +All values must be base64-encoded. + +In this setup, MinIO is deployed separately (for example, via Helm) and acts as a dependency for Milvus. KubeDB does not manage MinIO directly; it only uses the credentials provided through the secret to connect to the object storage. + +### spec.metaStorage + +`spec.metaStorage` defines how the metadata store (etcd) used by Milvus is configured. Milvus relies on etcd to manage cluster metadata and coordination. The etcd operator must be installed and running in the user cluster. + +- `externallyManaged` indicates whether the etcd cluster is managed outside of KubeDB. + - If true, users must provide the etcd endpoints. + - If false, KubeDB will create and manage an EtcdCluster resource. + +- `endpoints` is the list of etcd client endpoints and required when externallyManaged: true. +- `size (optional)` is the number of etcd nodes to provision and used only when externallyManaged: false. +- `storageType (optional)` defines storage behavior (e.g., durable or ephemeral). +- `storage (optional)` specifies the PersistentVolume configuration for etcd when managed by KubeDB. + +### spec.authSecret + +`spec.authSecret` is an optional field that points to a Secret used to hold credentials for `milvus` root user. If not set, KubeDB operator creates a new Secret `{milvus-object-name}-auth` for storing the password for `root` user for each Milvus object. + +We can use this field in 3 mode. +1. Using an external secret. In this case, You need to create an auth secret first with required fields, then specify the secret name when creating the Milvus object using `spec.authSecret.name` & set `spec.authSecret.externallyManaged` to true. +```yaml +authSecret: + name: + externallyManaged: true +``` + +2. Specifying the secret name only. In this case, You need to specify the secret name when creating the Milvus object using `spec.authSecret.name`. `externallyManaged` is by default false. +```yaml +authSecret: + name: +``` + +3. Let KubeDB do everything for you. In this case, no work for you. + +AuthSecret contains a `user` key and a `password` key which contains the `username` and `password` respectively for Milvus `root` user. + +Example: + +```bash +$ kubectl create secret generic milvus-auth -n demo \ +--from-literal=username=root \ +--from-literal=password=6q8u_2jMOW-OOZXk +secret "milvus-auth" created +``` + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: milvus-auth + namespace: kubedb +type: kubernetes.io/basic-auth +stringData: + username: "root" + password: "Milvus" +``` + +Secrets provided by users are not managed by KubeDB, and therefore, won't be modified or garbage collected by the KubeDB operator (version 0.13.0 and higher). + +### spec.configuration +`spec.configuration` is an optional field that specifies custom configuration for Milvus cluster. It has the following fields: +- `configuration.secretName` is an optional field that specifies the name of the secret that holds custom configuration files for Milvus cluster. +- `configuration.inline` is an optional field that allows you to provide custom configuration directly in the Milvus object. + +```yaml +configuration: + secretName: my-release-user-config + inline: + milvus.yaml: | + log: + level: info + file: + maxAge: 30 +``` + +### spec.topology + +`spec.topology` defines the deployment topology for Milvus. It supports both **Standalone** and **Distributed** (cluster) modes. + +### spec.topology.mode + +Specifies the deployment mode of Milvus. + +- **`Standalone`**: Runs Milvus as a single-node deployment. All components run inside a single pod. +- **`Distributed`**: Runs Milvus as a multi-component distributed cluster. + +### spec.topology.standalone + +```yaml +topology: + mode: Standalone +``` +When `mode: Standalone` is used: +- All Milvus components run in a single unified deployment. +- No separate component configuration (like datanode, proxy, etc.) is required. +- KubeDB manages all internal services automatically + +### spec.topology.distributed + +`distributed` contains the configuration for all Milvus components in distributed mode. + +#### spec.topology.mixcoord + +`mixcoord` is responsible for coordinating metadata and internal cluster orchestration. + +- **replicas**: Number of mixcoord pods (default: `1`) +- **podTemplate**: Custom resource requests/limits and pod-level configuration + + +#### spec.topology.distributed.datanode + +`datanode` handles data ingestion and persistence into storage. + +- **replicas**: Number of datanode pods (default: `1`) +- **podTemplate**: Resource configuration for each datanode pod + +#### spec.topology.distributed.proxy + +`proxy` is the entry point for client requests. + +- **replicas**: Number of proxy pods (default: `1`) +- **podTemplate**: Resource configuration for proxy pods + +#### spec.topology.distributed.querynode + +`querynode` executes search and query operations on vector data. + +- **replicas**: Number of querynode pods (default: `1`) +- **podTemplate**: Resource configuration for query execution workload + + +#### spec.topology.distributed.streamingnode + +`streamingnode` handles real-time streaming ingestion. + +- **replicas**: Number of streamingnode pods (default: `1`) +- **podTemplate**: Resource configuration for streaming workloads + +Additional storage configuration for `streamingnode`: + +- **storageType**: Defines storage behavior (`Durable` or `Ephemeral`) +- **storage**: PVC specification used when `storageType` is `Durable` + +### spec..podTemplate + +KubeDB allows providing a template for database pod through `spec..podTemplate`. KubeDB operator will pass the information provided in `spec..podTemplate` to the PetSet created for Milvus cluster. + +KubeDB accept following fields to set in `spec..podTemplate:` + +- metadata: + - annotations (pod's annotation) + - labels (pod's labels) +- controller: + - annotations (petset's annotation) + - labels (petset's labels) +- spec: + - containers + - volumes + - podPlacementPolicy + - initContainers + - containers + - imagePullSecrets + - nodeSelector + - serviceAccountName + - schedulerName + - tolerations + - priorityClassName + - priority + - securityContext + +You can check out the full list [here](https://github.com/kmodules/offshoot-api/blob/master/api/v2/types.go#L26C1-L279C1). +Uses of some field of `spec..podTemplate` is described below, + +#### spec..podTemplate.spec.tolerations + +The `spec.podTemplate.spec.tolerations` is an optional field. This can be used to specify the pod's tolerations. + +#### spec..podTemplate.spec.volumes + +The `spec..podTemplate..volumes` is an optional field. This can be used to provide the list of volumes that can be mounted by containers belonging to the pod. + +#### spec..podTemplate.spec.podPlacementPolicy + +`spec.podTemplate.spec.podPlacementPolicy` is an optional field. This can be used to provide the reference of the `podPlacementPolicy`. `name` of the podPlacementPolicy is referred under this attribute. This will be used by our Petset controller to place the db pods throughout the region, zone & nodes according to the policy. It utilizes kubernetes affinity & podTopologySpreadContraints feature to do so. +```yaml +spec: + podPlacementPolicy: + name: default +``` + +#### spec..podTemplate.spec.nodeSelector + +`spec..podTemplate.spec.nodeSelector` is an optional field that specifies a map of key-value pairs. For the pod to be eligible to run on a node, the node must have each of the indicated key-value pairs as labels (it can have additional labels as well). To learn more, see [here](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector) . + +#### spec..podTemplate.spec.containers + +The `spec..podTemplate.spec.containers` can be used to provide the list containers and their configurations for to the database pod. some of the fields are described below, + +##### spec..podTemplate.spec.containers[].name +The `spec..podTemplate.spec.containers[].name` field used to specify the name of the container specified as a DNS_LABEL. Each container in a pod must have a unique name (DNS_LABEL). Cannot be updated. + +##### spec..podTemplate.spec.containers[].args +`spec..podTemplate.spec.containers[].args` is an optional field. This can be used to provide additional arguments to database installation. + +##### spec..podTemplate.spec.containers[].env + +`spec..podTemplate.spec.containers[].env` is an optional field that specifies the environment variables to pass to the Redis containers. + +##### spec..podTemplate.spec.containers[].resources + +`spec..podTemplate.spec.containers[].resources` is an optional field. This can be used to request compute resources required by containers of the database pods. To learn more, visit [here](http://kubernetes.io/docs/user-guide/compute-resources/). + +### spec.serviceTemplates + +You can also provide template for the services created by KubeDB operator for Milvus cluster through `spec.serviceTemplates`. This will allow you to set the type and other properties of the services. + +KubeDB allows following fields to set in `spec.serviceTemplates`: +- `alias` represents the identifier of the service. It has the following possible value: + - `stats` for is used for the `exporter` service identification. + +Milvus comes with one primary services used for client access (Standalone or Proxy in Distributed mode) and four component services for distributed mode (`mixcoord`, `datanode`, `querynode` and `streamingnode`). There are two options for providing serviceTemplates: +- To provide `serviceTemplates` for a specific service, the `serviceTemplates.ports.port` should be equal to the port of that service and `serviceTemplate` will be used for that particular service only. +- However, to provide a common `serviceTemplates`, `serviceTemplates.ports.port` should be empty. + +- metadata: + - labels + - annotations +- spec: + - type + - ports + - clusterIP + - externalIPs + - loadBalancerIP + - loadBalancerSourceRanges + - externalTrafficPolicy + - healthCheckNodePort + - sessionAffinityConfig + +See [here](https://github.com/kmodules/offshoot-api/blob/kubernetes-1.21.1/api/v1/types.go#L237) to understand these fields in detail. + +### spec.monitor + +Milvus managed by KubeDB can be monitored with Prometheus operator out-of-the-box. To learn more, +- [Monitor Apache Milvus with Prometheus operator](/docs/guides/milvus/monitoring/using-prometheus-operator.md) + +### spec.tls + +`spec.tls` defines the TLS configuration for securing Milvus communication. The KubeDB operator uses [cert-manager](https://cert-manager.io/) to issue and manage certificates. Currently, only **PKCS#8 encoded certificates** are supported. + +TLS in Milvus can be configured for: +- **External traffic** (client → Milvus) +- **Internal traffic** (inter-component communication) + +--- + +### Example + +```yaml +spec: + tls: + issuerRef: + apiGroup: cert-manager.io + kind: Issuer + name: milvus-issuer + external: + mode: mTLS + internal: + mode: TLS +``` +The `spec.tls` contains the following fields: + +- `tls.issuerRef` - is an `optional` field that references to the `Issuer` or `ClusterIssuer` custom resource object of [cert-manager](https://cert-manager.io/docs/concepts/issuer/). It is used to generate the necessary certificate secrets for Milvus. If the `issuerRef` is not specified, the operator creates a self-signed CA and also creates necessary certificate (valid: 365 days) secrets using that CA. + - `apiGroup` - is the group name of the resource that is being referenced. Currently, the only supported value is `cert-manager.io`. + - `kind` - is the type of resource that is being referenced. The supported values are `Issuer` and `ClusterIssuer`. + - `name` - is the name of the resource ( `Issuer` or `ClusterIssuer` ) that is being referenced. + + +- `tls.external` - `external` controls TLS for client-facing traffic (gRPC / REST). + - `TLS` - requires only the server certificate to encrypt communication between client and Milvus proxy. + - `mTLS` - requires both server and client certificates to enable mutual authentication between client and Milvus proxy. + + +- `tls.internal` - `internal` enables TLS for inter-component communication within the Milvus cluster. + - `TLS` - uses only server-side certificates to encrypt communication between internal Milvus components (e.g., proxy, querynode, datanode). + - `mTLS` - not supported for internal communication; internal traffic is generally secured using one-way TLS only. + + +- `tls.certificates` - is an `optional` field that specifies a list of certificate configurations used to configure the certificates. It has the following fields: + - `alias` - represents the identifier of the certificate. It has the following possible value: + - `server` - is used for the server certificate configuration. + - `client` - is used for the client certificate configuration. + + + - `secretName` - ( `string` | `"-alias-cert"` ) - specifies the k8s secret name that holds the certificates. + + + - `subject` - specifies an `X.509` distinguished name (DN). It has the following configurable fields: + - `organizations` ( `[]string` | `nil` ) - is a list of organization names. + - `organizationalUnits` ( `[]string` | `nil` ) - is a list of organization unit names. + - `countries` ( `[]string` | `nil` ) - is a list of country names (ie. Country Codes). + - `localities` ( `[]string` | `nil` ) - is a list of locality names. + - `provinces` ( `[]string` | `nil` ) - is a list of province names. + - `streetAddresses` ( `[]string` | `nil` ) - is a list of street addresses. + - `postalCodes` ( `[]string` | `nil` ) - is a list of postal codes. + - `serialNumber` ( `string` | `""` ) is a serial number. + + For more details, visit [here](https://golang.org/pkg/crypto/x509/pkix/#Name). + + + - `duration` ( `string` | `""` ) - is the period during which the certificate is valid. A duration string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix, such as `"300m"`, `"1.5h"` or `"20h45m"`. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". + - `renewBefore` ( `string` | `""` ) - is a specifiable time before expiration duration. + - `dnsNames` ( `[]string` | `nil` ) - is a list of subject alt names. + - `ipAddresses` ( `[]string` | `nil` ) - is a list of IP addresses. + - `uris` ( `[]string` | `nil` ) - is a list of URI Subject Alternative Names. + - `emailAddresses` ( `[]string` | `nil` ) - is a list of email Subject Alternative Names. + +### spec.deletionPolicy + +`deletionPolicy` gives flexibility whether to `nullify`(reject) the delete operation of `Milvus` crd or which resources KubeDB should keep or delete when you delete `Milvus` crd. KubeDB provides following four deletion policies: + +- DoNotTerminate +- WipeOut +- Halt +- Delete + +When `deletionPolicy` is `DoNotTerminate`, KubeDB takes advantage of `ValidationWebhook` feature in Kubernetes 1.9.0 or later clusters to implement `DoNotTerminate` feature. If admission webhook is enabled, `DoNotTerminate` prevents users from deleting the database as long as the `spec.deletionPolicy` is set to `DoNotTerminate`. + +> For more details you can visit [here](https://appscode.com/blog/post/deletion-policy/) + +### spec.healthChecker +It defines the attributes for the health checker. +- `spec.healthChecker.periodSeconds` specifies how often to perform the health check. +- `spec.healthChecker.timeoutSeconds` specifies the number of seconds after which the probe times out. +- `spec.healthChecker.failureThreshold` specifies minimum consecutive failures for the healthChecker to be considered failed. +- `spec.healthChecker.disableWriteCheck` specifies whether to disable the writeCheck or not. + +Know details about KubeDB Health checking from this [blog post](https://appscode.com/blog/post/kubedb-health-checker/). + +## Next Steps + +- Learn how to use KubeDB to run Apache Milvus cluster [here](/docs/guides/milvus/README.md). +- Deploy [dedicated topology cluster](/docs/guides/milvus/clustering/guide/index.md) for Apache Milvus +- Monitor your Milvus cluster with KubeDB using [`out-of-the-box` Prometheus operator](/docs/guides/milvus/monitoring/using-prometheus-operator.md). +- Detail concepts of [MilvusVersion object](/docs/guides/milvus/concepts/milvusversion.md). + +[//]: # (- Learn to use KubeDB managed Milvus objects using [CLIs](/docs/guides/milvus/cli/cli.md).) +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). \ No newline at end of file diff --git a/docs/guides/milvus/configuration/_index.md b/docs/guides/milvus/configuration/_index.md new file mode 100644 index 000000000..0224bf33d --- /dev/null +++ b/docs/guides/milvus/configuration/_index.md @@ -0,0 +1,10 @@ +--- +title: Run Milvus with Custom Configuration +menu: + docs_{{ .version }}: + identifier: milvus-configuration + name: Custom Configuration + parent: milvus-guides + weight: 130 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/milvus/configuration/using-config-file.md b/docs/guides/milvus/configuration/using-config-file.md new file mode 100644 index 000000000..1810929b3 --- /dev/null +++ b/docs/guides/milvus/configuration/using-config-file.md @@ -0,0 +1,137 @@ +--- +title: Run Milvus with Custom Configuration +menu: + docs_{{ .version }}: + identifier: milvus-using-config-file + name: Config File + parent: milvus-configuration + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Using Custom Configuration File + +KubeDB supports providing custom configuration for Milvus. This tutorial will show you how to use KubeDB to run Milvus with custom configuration. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). + +- Now, install KubeDB cli on your workstation and KubeDB operator in your cluster following the steps [here](/docs/setup/README.md). + +- To keep things isolated, this tutorial uses a separate namespace called `demo` throughout this tutorial. + +```bash +$ kubectl create ns demo +namespace/demo created +``` + +> Note: YAML files used in this tutorial are stored in [docs/examples/milvus](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/examples/milvus) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). + +## Overview + +Milvus supports configuration via the `milvus.yaml` file. KubeDB takes advantage of `spec.configuration.secretName` to allow users to provide a custom `milvus.yaml` without mounting any volume into the Pod. The operator reads this Secret internally and applies the configuration automatically. + +In this tutorial, we will configure `queryNode.gracefulTime` and `dataNode.segment.maxSize` parameters. + +## Custom Configuration + +At first, let's create a custom `milvus.yaml` file: + +```yaml +queryNode: + gracefulTime: 5000 + +dataNode: + segment: + maxSize: 512 +``` + +Now, create a Secret with this configuration file. + +```bash +$ kubectl create secret generic -n demo milvus-configuration \ + --from-file=milvus.yaml=./milvus.yaml +secret/milvus-configuration created +``` + +Verify the Secret has the configuration file. + +```bash +$ kubectl get secret -n demo milvus-configuration -o yaml +apiVersion: v1 +data: + milvus.yaml: +kind: Secret +metadata: + name: milvus-configuration + namespace: demo +``` + +Now, create Milvus CRD specifying `spec.configuration.secretName` field. + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: Milvus +metadata: + name: custom-milvus + namespace: demo +spec: + version: "2.6.11" + objectStorage: + configSecret: + name: "my-release-minio" + configuration: + secretName: milvus-configuration + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + deletionPolicy: WipeOut +``` + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/milvus/configuration/milvus-configuration.yaml +milvus.kubedb.com/custom-milvus created +``` + +Now, wait for the Milvus to be ready. + +```bash +$ kubectl get milvus -n demo custom-milvus +NAME VERSION STATUS AGE +custom-milvus 2.6.11 Ready 3m +``` + +Check that the pod is running: + +```bash +$ kubectl get pod -n demo custom-milvus-0 +NAME READY STATUS RESTARTS AGE +custom-milvus-0 1/1 Running 0 3m +``` + +Now, we will verify the configuration has been applied. We will `exec` into the pod and check the configuration file. + +```bash +$ kubectl exec -it -n demo custom-milvus-0 -- cat /milvus/configs/milvus.yaml | grep -A 5 queryNode +queryNode: + gracefulTime: 5000 +``` + +## Cleaning up + +To cleanup the Kubernetes resources created by this tutorial, run: + +```bash +kubectl patch -n demo milvus/custom-milvus -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +kubectl delete -n demo milvus/custom-milvus + +kubectl delete -n demo secret milvus-configuration +kubectl delete ns demo +``` diff --git a/docs/guides/milvus/custom-rbac/_index.md b/docs/guides/milvus/custom-rbac/_index.md new file mode 100644 index 000000000..d322e1d6d --- /dev/null +++ b/docs/guides/milvus/custom-rbac/_index.md @@ -0,0 +1,10 @@ +--- +title: Run Milvus with Custom RBAC resources +menu: + docs_{{ .version }}: + identifier: milvus-custom-rbac + name: Custom RBAC + parent: milvus-guides + weight: 90 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/milvus/custom-rbac/using-custom-rbac.md b/docs/guides/milvus/custom-rbac/using-custom-rbac.md new file mode 100644 index 000000000..a5ee49c75 --- /dev/null +++ b/docs/guides/milvus/custom-rbac/using-custom-rbac.md @@ -0,0 +1,160 @@ +--- +title: Run Milvus with Custom RBAC resources +menu: + docs_{{ .version }}: + identifier: milvus-custom-rbac-quickstart + name: Custom RBAC + parent: milvus-custom-rbac + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Using Custom RBAC Resources + +KubeDB supports finer user control over role based access permissions provided to a Milvus instance. This tutorial will show you how to use KubeDB to run Milvus instance with custom RBAC resources. + +## Before You Begin + +At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). + +Now, install KubeDB cli on your workstation and KubeDB operator in your cluster following the steps [here](/docs/setup/README.md). + +To keep things isolated, this tutorial uses a separate namespace called `demo` throughout this tutorial. + +```bash +$ kubectl create ns demo +namespace/demo created +``` + +## Overview + +KubeDB allows users to provide custom RBAC resources, namely, `ServiceAccount`, `Role`, and `RoleBinding` for Milvus. This is provided via the `spec.podTemplate.spec.serviceAccountName` field in Milvus CRD. + +## Custom RBAC for Milvus + +At first, let's create a `Service Account` in `demo` namespace. + +```bash +$ kubectl create serviceaccount -n demo my-custom-serviceaccount +serviceaccount/my-custom-serviceaccount created +``` + +Now, we need to create a role that has necessary access permissions for the Milvus database named `quick-milvus`. + +```yaml +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: my-custom-role + namespace: demo +rules: +- apiGroups: + - apps + resourceNames: + - quick-milvus + resources: + - petsets + verbs: + - get +- apiGroups: + - kubedb.com + resourceNames: + - quick-milvus + resources: + - milvuses + verbs: + - get +- apiGroups: + - "" + resources: + - pods + verbs: + - list + - patch +- apiGroups: + - "" + resources: + - pods/exec + verbs: + - create +- apiGroups: + - "" + resources: + - configmaps + verbs: + - create + - get + - update +``` + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/milvus/custom-rbac/milvus-custom-role.yaml +role.rbac.authorization.k8s.io/my-custom-role created +``` + +Now create a `RoleBinding` to bind this `Role` with the already created service account. + +```bash +$ kubectl create rolebinding my-custom-rolebinding \ + --role=my-custom-role \ + --serviceaccount=demo:my-custom-serviceaccount \ + --namespace=demo +rolebinding.rbac.authorization.k8s.io/my-custom-rolebinding created +``` + +Now, create a Milvus CRD specifying `spec.podTemplate.spec.serviceAccountName` field to `my-custom-serviceaccount`. + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: Milvus +metadata: + name: quick-milvus + namespace: demo +spec: + version: "2.6.11" + objectStorage: + configSecret: + name: "my-release-minio" + storageType: Durable + podTemplate: + spec: + serviceAccountName: my-custom-serviceaccount + storage: + storageClassName: "standard" + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + deletionPolicy: WipeOut +``` + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/milvus/custom-rbac/milvus-custom-db.yaml +milvus.kubedb.com/quick-milvus created +``` + +Check that the pod is running: + +```bash +$ kubectl get pod -n demo quick-milvus-0 +NAME READY STATUS RESTARTS AGE +quick-milvus-0 1/1 Running 0 3m +``` + +## Cleaning up + +To cleanup the Kubernetes resources created by this tutorial, run: + +```bash +kubectl patch -n demo milvus/quick-milvus -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +kubectl delete -n demo milvus/quick-milvus + +kubectl delete -n demo serviceaccount my-custom-serviceaccount +kubectl delete -n demo role my-custom-role +kubectl delete -n demo rolebinding my-custom-rolebinding +kubectl delete ns demo +``` diff --git a/docs/guides/milvus/monitoring/_index.md b/docs/guides/milvus/monitoring/_index.md new file mode 100644 index 000000000..a1c83edfb --- /dev/null +++ b/docs/guides/milvus/monitoring/_index.md @@ -0,0 +1,10 @@ +--- +title: Milvus Monitoring +menu: + docs_{{ .version }}: + identifier: milvus-monitoring + name: Monitoring + parent: milvus-guides + weight: 20 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/milvus/monitoring/overview.md b/docs/guides/milvus/monitoring/overview.md new file mode 100644 index 000000000..248796e32 --- /dev/null +++ b/docs/guides/milvus/monitoring/overview.md @@ -0,0 +1,39 @@ +--- +title: Milvus Monitoring Overview +menu: + docs_{{ .version }}: + identifier: milvus-monitoring-overview + name: Overview + parent: milvus-monitoring + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +# Milvus Monitoring + +This guide shows how to enable and verify monitoring for Milvus deployments. + +## Before You Begin + +- Deploy Milvus first using the [quickstart guide](/docs/guides/milvus/quickstart/quickstart.md). +- Install Prometheus Operator or another compatible metrics collection stack. + +## Enable Monitoring + +Milvus supports monitoring integration through `spec.monitor`. + +- Configure monitoring agent and service monitor selectors. +- Validate scrape targets for proxy and distributed components. + +## Verify + +```bash +kubectl get milvus -n demo milvus-cluster -o yaml +kubectl get servicemonitor -A +``` + +## Next Steps + +- Review which Milvus components need dedicated scrape targets in distributed mode. +- Add dashboards for latency, indexing throughput, and node health. diff --git a/docs/guides/milvus/monitoring/using-prometheus-operator.md b/docs/guides/milvus/monitoring/using-prometheus-operator.md new file mode 100644 index 000000000..ac5487e9d --- /dev/null +++ b/docs/guides/milvus/monitoring/using-prometheus-operator.md @@ -0,0 +1,344 @@ +--- +title: Monitor Milvus using Prometheus Operator +menu: + docs_{{ .version }}: + identifier: milvus-using-prometheus-operator-monitoring + name: Prometheus Operator + parent: milvus-monitoring + weight: 30 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Monitoring Milvus using Prometheus Operator + +[Prometheus operator](https://github.com/prometheus-operator/prometheus-operator) provides simple and Kubernetes native way to deploy and configure Prometheus, Alertmanager and related monitoring components. This tutorial will show you how to use the Prometheus operator to monitor Milvus database deployed with KubeDB. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). + +- Install KubeDB operator in your cluster following the steps [here](/docs/setup/README.md). + +- To learn how Prometheus monitoring works with KubeDB in general, please visit [here](/docs/guides/milvus/monitoring/overview.md). + +- To keep Prometheus resources isolated, we are going to use a separate namespace called `monitoring` to deploy respective monitoring resources. We are going to deploy the database in `demo` namespace. + + ```bash + $ kubectl create ns monitoring + namespace/monitoring created + + $ kubectl create ns demo + namespace/demo created + ``` + +> Note: YAML files used in this tutorial are stored in [docs/examples/milvus](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/examples/milvus) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). + +## Find out required labels for ServiceMonitor + +We need to know the labels used to select `ServiceMonitor` by a `Prometheus` crd. We are going to provide these labels in `spec.monitor.prometheus.serviceMonitor.labels` field of Milvus crd so that KubeDB creates `ServiceMonitor` object accordingly. + +At first, let's find out the available Prometheus server in our cluster. + +```bash +$ kubectl get prometheus --all-namespaces +NAMESPACE NAME VERSION DESIRED READY RECONCILED AVAILABLE AGE +monitoring prometheus-kube-prometheus-prometheus v3.11.3 1 1 True True 25h +``` + +> If you don't have any Prometheus server running in your cluster, deploy one following the guide specified in **Before You Begin** section. + +Now, let's view the YAML of the available Prometheus server `prometheus-kube-prometheus-prometheus` in `monitoring` namespace. + +```bash +$ kubectl get prometheus -n monitoring prometheus-kube-prometheus-prometheus -oyaml +``` +```yaml +apiVersion: monitoring.coreos.com/v1 +kind: Prometheus +metadata: + annotations: + meta.helm.sh/release-name: prometheus + meta.helm.sh/release-namespace: monitoring + creationTimestamp: "2026-05-04T04:56:27Z" + generation: 1 + labels: + app: kube-prometheus-stack-prometheus + app.kubernetes.io/instance: prometheus + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/part-of: kube-prometheus-stack + app.kubernetes.io/version: 84.5.0 + chart: kube-prometheus-stack-84.5.0 + heritage: Helm + release: prometheus + name: prometheus-kube-prometheus-prometheus + namespace: monitoring + resourceVersion: "404630" + uid: 6cf79306-5f52-46e7-9342-b7447016823e +spec: + affinity: + podAntiAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - podAffinityTerm: + labelSelector: + matchExpressions: + - key: app.kubernetes.io/name + operator: In + values: + - prometheus + - key: app.kubernetes.io/instance + operator: In + values: + - prometheus-kube-prometheus-prometheus + topologyKey: kubernetes.io/hostname + weight: 100 + alerting: + alertmanagers: + - apiVersion: v2 + name: prometheus-kube-prometheus-alertmanager + namespace: monitoring + pathPrefix: / + port: http-web + automountServiceAccountToken: true + enableAdminAPI: false + enableOTLPReceiver: false + evaluationInterval: 30s + externalUrl: http://prometheus-kube-prometheus-prometheus.monitoring:9090 + hostNetwork: false + image: quay.io/prometheus/prometheus:v3.11.3 + imagePullPolicy: IfNotPresent + listenLocal: false + logFormat: logfmt + logLevel: info + paused: false + podMonitorNamespaceSelector: {} + podMonitorSelector: + matchLabels: + release: prometheus + portName: http-web + probeNamespaceSelector: {} + probeSelector: + matchLabels: + release: prometheus + replicas: 1 + retention: 10d + routePrefix: / + ruleNamespaceSelector: {} + ruleSelector: + matchLabels: + release: prometheus + scrapeConfigNamespaceSelector: {} + scrapeConfigSelector: + matchLabels: + release: prometheus + scrapeInterval: 30s + securityContext: + fsGroup: 2000 + runAsGroup: 2000 + runAsNonRoot: true + runAsUser: 1000 + seccompProfile: + type: RuntimeDefault + serviceAccountName: prometheus-kube-prometheus-prometheus + serviceMonitorNamespaceSelector: {} + serviceMonitorSelector: + matchLabels: + release: prometheus + shards: 1 + tsdb: + outOfOrderTimeWindow: 0s + version: v3.11.3 + walCompression: true +status: + availableReplicas: 1 + conditions: + - lastTransitionTime: "2026-05-04T04:57:09Z" + message: "" + observedGeneration: 1 + reason: "" + status: "True" + type: Available + - lastTransitionTime: "2026-05-04T04:56:37Z" + message: "" + observedGeneration: 1 + reason: "" + status: "True" + type: Reconciled + paused: false + replicas: 1 + selector: app.kubernetes.io/instance=prometheus-kube-prometheus-prometheus,app.kubernetes.io/managed-by=prometheus-operator,app.kubernetes.io/name=prometheus,operator.prometheus.io/name=prometheus-kube-prometheus-prometheus,prometheus=prometheus-kube-prometheus-prometheus + shardStatuses: + - availableReplicas: 1 + replicas: 1 + shardID: "0" + unavailableReplicas: 0 + updatedReplicas: 1 + shards: 1 + unavailableReplicas: 0 + updatedReplicas: 1 +``` +Notice the `spec.serviceMonitorSelector` section. Here, `release: prometheus` label is used to select `ServiceMonitor` CR. So, we are going to use this label in `spec.monitor.prometheus.labels` field of Milvus CR. + + +## Deploy Milvus with Monitoring Enabled + +At first, let's deploy a Milvus database with monitoring enabled. + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: Milvus +metadata: + name: coreos-prom-milvus + namespace: demo +spec: + version: "2.6.11" + objectStorage: + configSecret: + name: "my-release-minio" + topology: + mode: Distributed + deletionPolicy: WipeOut + monitor: + agent: prometheus.io/operator + prometheus: + serviceMonitor: + interval: 10s + labels: + release: prometheus + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi +``` + +Here, +- `spec.monitor.agent: prometheus.io/operator` indicates that we are going to monitor this server using Prometheus operator. +- `spec.monitor.prometheus.serviceMonitor.labels` specifies that KubeDB should create `ServiceMonitor` with these labels. +- `spec.monitor.prometheus.serviceMonitor.interval` indicates that the Prometheus should scrape metrics from this database with 10 seconds interval. + +Let's create the Milvus object that we have shown above, + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/milvus/monitoring/coreos-prom-milvus.yaml +milvus.kubedb.com/coreos-prom-milvus created +``` + +Now, wait for the database to go into `Ready` state. + +```bash +$ kubectl get milvus -n demo coreos-prom-milvus +NAME VERSION STATUS AGE +coreos-prom-milvus 2.6.11 Ready 2m +``` +KubeDB will create a separate stats service with name `{Milvus crd name}-stats` for monitoring purpose. + +```bash +$ kubectl get svc -n kubedb --selector="app.kubernetes.io/instance=coreos-prom-milvus" +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +coreos-prom-milvus-stats ClusterIP 10.43.113.92 9091/TCP 50m +``` +Here, `coreos-prom-milvus-stats` service has been created for monitoring purpose. +Let's describe this stats service. + +```bash +$ kubectl describe svc -n demo coreos-prom-milvus-stats +``` +```yaml +Name: coreos-prom-milvus-stats +Namespace: kubedb +Labels: app.kubernetes.io/component=database + app.kubernetes.io/instance=coreos-prom-milvus-stats + app.kubernetes.io/managed-by=kubedb.com + app.kubernetes.io/name=milvuses.kubedb.com + kubedb.com/role=stats +Annotations: monitoring.appscode.com/agent: prometheus.io/operator +Selector: app.kubernetes.io/instance=coreos-prom-milvus-stats,app.kubernetes.io/managed-by=kubedb.com,app.kubernetes.io/name=milvuses.kubedb.com +Type: ClusterIP +IP Family Policy: SingleStack +IP Families: IPv4 +IP: 10.43.113.92 +IPs: 10.43.113.92 +Port: metrics 9091/TCP +TargetPort: metrics/TCP +Endpoints: 10.42.0.102:9091,10.42.0.107:9091,10.42.0.111:9091 + 2 more... +Session Affinity: None +Events: +``` +Notice the `Labels` and `Port` fields. `ServiceMonitor` will use this information to target its endpoints. + +KubeDB will also create a `ServiceMonitor` crd in `demo` namespace that select the endpoints of `coreos-prom-milvus-stats` service. Verify that the `ServiceMonitor` crd has been created. + +```bash +$ kubectl get servicemonitor -n demo +NAME AGE +coreos-prom-milvus-stats 2m +``` + + +Let's verify the `ServiceMonitor` has the right label to be discovered by Prometheus: + +```bash +$ kubectl get servicemonitor -n demo coreos-prom-milvus-stats -o yaml +apiVersion: monitoring.coreos.com/v1 +kind: ServiceMonitor +metadata: + labels: + release: prometheus + name: coreos-prom-milvus-stats + namespace: demo +spec: + endpoints: + - honorLabels: true + interval: 10s + path: /metrics + port: metrics + namespaceSelector: + matchNames: + - demo + selector: + matchLabels: + app.kubernetes.io/instance: coreos-prom-milvus +``` +Notice that the `ServiceMonitor` has label `release: prometheus` that we had specified in Milvus crd. + +Also notice that the `ServiceMonitor` has selector which match the labels we have seen in the `coreos-prom-milvus-stats` service. It also, target the `metrics` port that we have seen in the stats service. + +## Verify Monitoring Metrics + +At first, let's find out the respective Prometheus pod for `prometheus` Prometheus server. + +```bash +$ kubectl get pod -n monitoring -l=app.kubernetes.io/name=prometheus +NAME READY STATUS RESTARTS AGE +prometheus-prometheus-kube-prometheus-prometheus-0 2/2 Running 0 27h +``` + +Prometheus server is listening to port `9090` of `prometheus-prometheus-kube-prometheus-prometheus-0` pod. We are going to use [port forwarding](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/) to access Prometheus dashboard. + +Run following command on a separate terminal to forward the port 9090 of `prometheus-prometheus-kube-prometheus-prometheus-0` pod, + +```bash +$ kubectl port-forward -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 9090 +Forwarding from 127.0.0.1:9090 -> 9090 +Forwarding from [::1]:9090 -> 9090 +``` + +Now, we can access the dashboard at `localhost:9090`. Open [http://localhost:9090](http://localhost:9090) in your browser. You should see `metrics` endpoint of `coreos-prom-milvus-stats` service as one of the targets. + +Check the `endpoint` and `service` labels marked by the red rectangles. It verifies that the target is our expected database. Now, you can view the collected metrics and create a graph from homepage of this Prometheus dashboard. You can also use this Prometheus server as data source for [Grafana](https://grafana.com/) and create a beautiful dashboard with collected metrics. + +## Cleaning up + +To cleanup the Kubernetes resources created by this tutorial, run: + +```bash +kubectl patch -n demo milvus/coreos-prom-milvus -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +kubectl delete -n demo milvus/coreos-prom-milvus + +kubectl delete ns demo +kubectl delete ns monitoring +``` diff --git a/docs/guides/milvus/ops-request/_index.md b/docs/guides/milvus/ops-request/_index.md new file mode 100644 index 000000000..9d0b5d697 --- /dev/null +++ b/docs/guides/milvus/ops-request/_index.md @@ -0,0 +1,10 @@ +--- +title: Ops Request +menu: + docs_{{ .version }}: + identifier: milvus-ops-request + name: Ops Request + parent: milvus-guides + weight: 30 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/milvus/ops-request/overview.md b/docs/guides/milvus/ops-request/overview.md new file mode 100644 index 000000000..ae0b7e728 --- /dev/null +++ b/docs/guides/milvus/ops-request/overview.md @@ -0,0 +1,29 @@ +--- +title: Milvus Ops Request Overview +menu: + docs_{{ .version }}: + identifier: milvus-ops-request-overview + name: Overview + parent: milvus-ops-request + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +# Milvus Ops Request + +This page tracks Milvus operations supported by the new database matrix used for these guides. + +## Before You Begin + +- Deploy Milvus first using the [quickstart guide](/docs/guides/milvus/quickstart/quickstart.md). +- Confirm the current support list in [new_db.md](/new_db.md) for your release. + +## Current Status + +Based on [new_db.md](/new_db.md), no Milvus ops request types are listed yet. + +## What to Check Next + +- Watch future KubeDB releases for Milvus `OpsRequest` support. +- Use the TLS and monitoring guides for common day-2 platform tasks. diff --git a/docs/guides/milvus/private-registry/_index.md b/docs/guides/milvus/private-registry/_index.md new file mode 100644 index 000000000..c202e6450 --- /dev/null +++ b/docs/guides/milvus/private-registry/_index.md @@ -0,0 +1,10 @@ +--- +title: Run Milvus using Private Registry +menu: + docs_{{ .version }}: + identifier: milvus-private-registry + name: Private Registry + parent: milvus-guides + weight: 120 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/milvus/private-registry/using-private-registry.md b/docs/guides/milvus/private-registry/using-private-registry.md new file mode 100644 index 000000000..3ebbb5ab4 --- /dev/null +++ b/docs/guides/milvus/private-registry/using-private-registry.md @@ -0,0 +1,126 @@ +--- +title: Run Milvus using Private Registry +menu: + docs_{{ .version }}: + identifier: milvus-using-private-registry + name: Quickstart + parent: milvus-private-registry + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Using Private Docker Registry + +KubeDB supports using private Docker registries. This tutorial will show you how to run KubeDB managed Milvus database using private Docker images. + +## Before You Begin + +At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). + +To keep things isolated, this tutorial uses a separate namespace called `demo` throughout this tutorial. + +```bash +$ kubectl create ns demo +namespace/demo created +``` + +## Prepare Private Docker Registry + +- You will also need a docker private [registry](https://docs.docker.com/registry/) or [private repository](https://docs.docker.com/docker-hub/repos/#private-repositories). + +- Push the required images from KubeDB's [Docker hub account](https://hub.docker.com/r/kubedb/) into your private registry. For Milvus, push the `DB_IMAGE` of the following MilvusVersions, where `deprecated` is not true. + + ```bash + $ kubectl get milvusversions -o=custom-columns=NAME:.metadata.name,VERSION:.spec.version,DB_IMAGE:.spec.db.image,DEPRECATED:.spec.deprecated + NAME VERSION DB_IMAGE DEPRECATED + 2.4.0 2.4.0 kubedb/milvus:2.4.0 + ``` + +## Create ImagePullSecret + +Run the following command to create an image pull secret for your private Docker registry: + +```bash +$ kubectl create secret docker-registry -n demo myregistrykey \ + --docker-server=DOCKER_REGISTRY_SERVER \ + --docker-username=DOCKER_USER \ + --docker-email=DOCKER_EMAIL \ + --docker-password=DOCKER_PASSWORD +secret/myregistrykey created +``` + +## Install KubeDB Operator + +Follow the steps to [install KubeDB operator](/docs/setup/README.md) properly in cluster so that it points to the DOCKER_REGISTRY you wish to pull images from. + +## Create MilvusVersion CRD + +Create a MilvusVersion CRD specifying images from your private registry. Replace `PRIVATE_REGISTRY` with your private registry. + +```yaml +apiVersion: catalog.kubedb.com/v1alpha1 +kind: MilvusVersion +metadata: + name: "2.4.0" +spec: + db: + image: PRIVATE_REGISTRY/milvus:2.4.0 + version: "2.4.0" +``` + +```bash +$ kubectl apply -f pvt-milvusversion.yaml +milvusversion.catalog.kubedb.com/2.4.0 created +``` + +## Deploy Milvus from Private Registry + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: Milvus +metadata: + name: pvt-reg-milvus + namespace: demo +spec: + version: "2.4.0" + storageType: Durable + storage: + storageClassName: "standard" + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + podTemplate: + spec: + imagePullSecrets: + - name: myregistrykey + deletionPolicy: WipeOut +``` + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/milvus/private-registry/pvt-reg-milvus.yaml +milvus.kubedb.com/pvt-reg-milvus created +``` + +Check that the Milvus is in Running state: + +```bash +$ kubectl get pods -n demo --selector="app.kubernetes.io/instance=pvt-reg-milvus" +NAME READY STATUS RESTARTS AGE +pvt-reg-milvus-0 1/1 Running 0 3m +``` + +## Cleaning up + +To cleanup the Kubernetes resources created by this tutorial, run: + +```bash +kubectl patch -n demo milvus/pvt-reg-milvus -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +kubectl delete -n demo milvus/pvt-reg-milvus + +kubectl delete ns demo +``` diff --git a/docs/guides/milvus/quickstart/_index.md b/docs/guides/milvus/quickstart/_index.md new file mode 100644 index 000000000..753aefff9 --- /dev/null +++ b/docs/guides/milvus/quickstart/_index.md @@ -0,0 +1,10 @@ +--- +title: Milvus Quickstart +menu: + docs_{{ .version }}: + identifier: milvus-quickstart + name: Quickstart + parent: milvus-guides + weight: 15 +menu_name: docs_{{ .version }} +--- \ No newline at end of file diff --git a/docs/guides/milvus/quickstart/quickstart.md b/docs/guides/milvus/quickstart/quickstart.md new file mode 100644 index 000000000..9078abdab --- /dev/null +++ b/docs/guides/milvus/quickstart/quickstart.md @@ -0,0 +1,158 @@ +--- +title: Milvus Quickstart +menu: + docs_{{ .version }}: + identifier: milvus-quickstart-overview + name: Overview + parent: milvus-quickstart + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Milvus QuickStart + +This tutorial shows how to run a Milvus database with KubeDB. + +

+  lifecycle +

+ +> Note: YAML files used in this tutorial are stored in [docs/examples/milvus/quickstart](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/examples/milvus/quickstart). + +## Before You Begin + +At first, you need to have a Kubernetes cluster, and the `kubectl` command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). + +Now, install KubeDB cli on your workstation and KubeDB operator in your cluster following the steps [here](/docs/setup/README.md) and make sure to include the flags `--set global.featureGates.Milvus=true` to ensure **Milvus CRD**. + +To keep things isolated, this tutorial uses a separate namespace called `demo` throughout this tutorial. + +```bash +$ kubectl create namespace demo +namespace/demo created + +$ kubectl get namespace +NAME STATUS AGE +demo Active 9s +``` + +## Check Available MilvusVersion + +When you install the KubeDB operator, it registers a CRD named [MilvusVersion](/docs/guides/milvus/concepts/catalog.md). The installation process comes with a set of tested MilvusVersion objects. Let's check available MilvusVersions by, + +```bash +$ kubectl get milvusversions +NAME VERSION DB_IMAGE DEPRECATED AGE +2.6.11 2.6.11 ghcr.io/appscode-images/milvus:2.6.11 6d2h +2.6.7 2.6.7 ghcr.io/appscode-images/milvus:2.6.7 6d2h +2.6.9 2.6.9 ghcr.io/appscode-images/milvus:2.6.9 6d2h +``` + +Notice the `DEPRECATED` column. Here, `true` means that this MilvusVersion is deprecated for the current KubeDB version. KubeDB will not work for deprecated MilvusVersion. You can also use the short from `mvversion` to check available MilvusVersions. + +In this tutorial, we will use `2.6.11` MilvusVersion CR to create a Milvus cluster. + +## Get External Dependencies Ready + +### Object Storage + +One of the external dependency of Milvus is object storage where the segments are stored. It is a storage mechanism that Milvus does not provide. **S3-compatible storage** (like **Minio**) are generally convenient options for object storage. + +In this tutorial, we will run a `minio-server` as object storage in our local `kind` cluster using `minio-operator` and create a bucket named `milvus` in it, which the deployed milvus database will use. + +```bash + +$ helm repo add minio https://operator.min.io/ +$ helm repo update minio +$ helm upgrade --install --namespace "minio-operator" --create-namespace "minio-operator" minio/operator --set operator.replicaCount=1 + +$ helm upgrade --install --namespace "demo" --create-namespace milvus-minio minio/tenant \ +--set tenant.pools[0].servers=1 \ +--set tenant.pools[0].volumesPerServer=1 \ +--set tenant.pools[0].size=1Gi \ +--set tenant.certificate.requestAutoCert=false \ +--set tenant.buckets[0].name="milvus" \ +--set tenant.pools[0].name="default" + +``` + +Now we need to create a `Secret` named `my-release-minio`. It contains the necessary connection information using which the milvus database will connect to the object storage. + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: my-release-minio + namespace: demo +stringData: + milvus.storage.type: "s3" + milvus.storage.bucket: "milvus" + milvus.storage.baseKey: "milvus/segments" + milvus.s3.accessKey: "minio" + milvus.s3.secretKey: "minio123" + milvus.s3.protocol: "http" + milvus.s3.enablePathStyleAccess: "true" + milvus.s3.endpoint.signingRegion: "us-east-1" + milvus.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/" +``` + +Let’s create the `deep-storage-config` Secret shown above: + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/milvus/quickstart/deep-storage-config.yaml +secret/deep-storage-config created +``` + +You can also use options like **Amazon S3**, **Google Cloud Storage**, **Azure Blob Storage** or **HDFS** and create a connection information `Secret` like this, and you are good to go. + +## Create a Milvus Database + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: Milvus +metadata: + name: milvus-cluster + namespace: demo +spec: + version: "2.6.11" + objectStorage: + configSecret: + name: "my-release-minio" + topology: + mode: Distributed + distributed: + mixcoord: + replicas: 2 + storageType: Durable + storage: + accessModes: + - ReadWriteOnce + storageClassName: local-path + resources: + requests: + storage: 10Gi +``` + +```bash +kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/milvus/quickstart/distributed.yaml +kubectl get milvus -n demo milvus-cluster -w +``` + +## Verify Milvus Database + +```bash +kubectl get milvus -n demo +kubectl describe milvus -n demo milvus-cluster +``` + +When `status.phase` becomes `Ready`, the Milvus deployment is ready to serve vector search traffic. + +## Cleaning up + +```bash +kubectl delete milvus -n demo milvus-cluster +kubectl delete ns demo +``` \ No newline at end of file diff --git a/docs/guides/milvus/tls/_index.md b/docs/guides/milvus/tls/_index.md new file mode 100644 index 000000000..0a6bca5d2 --- /dev/null +++ b/docs/guides/milvus/tls/_index.md @@ -0,0 +1,10 @@ +--- +title: Milvus TLS +menu: + docs_{{ .version }}: + identifier: milvus-tls + name: TLS + parent: milvus-guides + weight: 25 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/milvus/tls/overview.md b/docs/guides/milvus/tls/overview.md new file mode 100644 index 000000000..979dff780 --- /dev/null +++ b/docs/guides/milvus/tls/overview.md @@ -0,0 +1,39 @@ +--- +title: Milvus TLS Overview +menu: + docs_{{ .version }}: + identifier: milvus-tls-overview + name: Overview + parent: milvus-tls + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +# Milvus TLS + +This guide shows the main TLS considerations for a Milvus deployment. + +## Before You Begin + +- Install `cert-manager` in your cluster. +- Deploy Milvus first using the [quickstart guide](/docs/guides/milvus/quickstart/quickstart.md). + +## Configure TLS + +Milvus TLS setup is managed with cert-manager and KubeDB TLS settings. + +- Use a trusted issuer for server certificates. +- Rotate certificates in a controlled maintenance window. + +## Verify + +```bash +kubectl get milvus -n demo milvus-cluster -o yaml +kubectl get secret -n demo +``` + +## Next Steps + +- Test client connectivity with TLS enabled before routing production traffic. +- Document the certificate rotation process for your team. diff --git a/docs/images/milvus/milvus-lifecycle.png b/docs/images/milvus/milvus-lifecycle.png new file mode 100644 index 000000000..45aff469d Binary files /dev/null and b/docs/images/milvus/milvus-lifecycle.png differ