Grok_connection server is failed

seritaka.gmail.com · July 21, 2023, 12:49pm

Dear all,

I’m newbie of datagrok.
I updated datagrok yesterday. The process seems went well but after the updating, grok_connection service status is failed.
How can I fix the issue?
Any suggestions or advices are greatly appreciated.
Thanks,
Taka

spodolskaya · July 21, 2023, 12:54pm

Hello Taka,

Could you kindly provide the logs for the service? If you started the service using docker-compose, you can retrieve the logs by entering the following command:
docker compose -f docker/localhost.docker-compose.yaml --project-name datagrok --profile all logs grok_connect

seritaka.gmail.com · July 21, 2023, 1:14pm

Hello spodolskaya,

Thanks for your prompt response.
This is the log.

datagrok-grok_connect-1 | 10:42:31.497 [main] INFO grok_connect.GrokConnect - grok_connect - version: 2.1.3
datagrok-grok_connect-1 | 10:42:31.502 [main] INFO grok_connect.GrokConnect - Grok Connect initializing
datagrok-grok_connect-1 | 10:42:31.504 [main] INFO grok_connect.GrokConnect - Memory: free: 3799351808(99.52%), used: 18513408
datagrok-grok_connect-1 | 10:42:31.504 [main] DEBUG grok_connect.utils.ProviderManager - Initializing providers HashMap
datagrok-grok_connect-1 | 10:42:31.662 [main] INFO grok_connect.GrokConnect - grok_connect with Hikari pool
datagrok-grok_connect-1 | 10:42:31.663 [main] INFO grok_connect.GrokConnect - grok_connect: Running on http://localhost:1234
datagrok-grok_connect-1 | 10:42:31.663 [main] INFO grok_connect.GrokConnect - grok_connect: Connectors: [Cassandra, Access, Firebird, Hive2, Neptune, SAP HANA, Neo4j, Snowflake, SQLite, MySQL, Postgres, Oracle, Denodo, BigQuery, ClickHouse, DynamoDB, MariaDB, Impala, Teradata, Redshift, HBase, MS SQL, Hive, DB2, Vertica, PI, MongoDB, Athena, Virtuoso]
datagrok-grok_connect-1 | 10:54:22.744 [main] INFO grok_connect.GrokConnect - grok_connect - version: 2.1.3
datagrok-grok_connect-1 | 10:54:22.746 [main] INFO grok_connect.GrokConnect - Grok Connect initializing
datagrok-grok_connect-1 | 10:54:22.748 [main] INFO grok_connect.GrokConnect - Memory: free: 3799351808(99.52%), used: 18513408
datagrok-grok_connect-1 | 10:54:22.748 [main] DEBUG grok_connect.utils.ProviderManager - Initializing providers HashMap
datagrok-grok_connect-1 | 10:54:22.894 [main] INFO grok_connect.GrokConnect - grok_connect with Hikari pool
datagrok-grok_connect-1 | 10:54:22.894 [main] INFO grok_connect.GrokConnect - grok_connect: Running on http://localhost:1234
datagrok-grok_connect-1 | 10:54:22.894 [main] INFO grok_connect.GrokConnect - grok_connect: Connectors: [Cassandra, Access, Firebird, Hive2, Neptune, SAP HANA, Neo4j, Snowflake, SQLite, MySQL, Postgres, Oracle, Denodo, BigQuery, ClickHouse, DynamoDB, MariaDB, Impala, Teradata, Redshift, HBase, MS SQL, Hive, DB2, Vertica, PI, MongoDB, Athena, Virtuoso]
datagrok-grok_connect-1 | 10:59:30.310 [main] INFO grok_connect.GrokConnect - grok_connect - version: 2.1.3
datagrok-grok_connect-1 | 10:59:30.311 [main] INFO grok_connect.GrokConnect - Grok Connect initializing
datagrok-grok_connect-1 | 10:59:30.313 [main] INFO grok_connect.GrokConnect - Memory: free: 3799351808(99.52%), used: 18513408
datagrok-grok_connect-1 | 10:59:30.313 [main] DEBUG grok_connect.utils.ProviderManager - Initializing providers HashMap
datagrok-grok_connect-1 | 10:59:30.382 [main] INFO grok_connect.GrokConnect - grok_connect with Hikari pool
datagrok-grok_connect-1 | 10:59:30.382 [main] INFO grok_connect.GrokConnect - grok_connect: Running on http://localhost:1234
datagrok-grok_connect-1 | 10:59:30.383 [main] INFO grok_connect.GrokConnect - grok_connect: Connectors: [Cassandra, Access, Firebird, Hive2, Neptune, SAP HANA, Neo4j, Snowflake, SQLite, MySQL, Postgres, Oracle, Denodo, BigQuery, ClickHouse, DynamoDB, MariaDB, Impala, Teradata, Redshift, HBase, MS SQL, Hive, DB2, Vertica, PI, MongoDB, Athena, Virtuoso]

Thanks,
Taka

spodolskaya · July 21, 2023, 1:26pm

It appears that the container is restarting, but the logs do not indicate a specific reason for this. Could you please share the output of the following commands with us?

docker ps -a
docker ps -a | grep 'connect' | awk '{print $1}' | xargs docker logs --details --timestamps

Thank you,
Sofia

seritaka.gmail.com · July 21, 2023, 1:32pm

Thaks for prompt reply. Here is an out puts of the commands which you suggested.
1st one,

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
524e3d4d14bf datagrok/datagrok:latest “/bin/sh -c 'bash ${…” 3 hours ago Up 3 hours 1234/tcp, 8082/tcp, 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp datagrok-datagrok-1
4255eeaaed21 datagrok/cvm_nginx:latest “/bin/sh -c '/etc/ng…” 3 hours ago Up 3 hours 80/tcp, 0.0.0.0:8090->8090/tcp, :::8090->8090/tcp datagrok-cvm_nginx-1
ee370b6be991 datagrok/jupyter_kernel_gateway:latest “/bin/bash -o pipefa…” 3 hours ago Up 3 hours 5005/tcp, 8082/tcp, 8888/tcp datagrok-jupyter_kernel_gateway-1
14d1f54721b6 datagrok/grok_spawner:latest “/docker-entrypoint.…” 3 hours ago Up 3 hours 2375-2376/tcp, 0.0.0.0:8000->8000/tcp, :::8000->8000/tcp datagrok-grok_spawner-1
2996c7520a3d datagrok/jupyter_notebook:latest “/bin/bash -o pipefa…” 3 hours ago Up 3 hours 5005/tcp, 8889/tcp datagrok-jupyter_notebook-1
7a4ffc38ef6e datagrok/h2o:latest “/bin/bash -o pipefa…” 3 hours ago Up 3 hours 0.0.0.0:5005->5005/tcp, :::5005->5005/tcp, 0.0.0.0:54321->54321/tcp, :::54321->54321/tcp datagrok-h2o-1
1625b4941f13 datagrok/grok_compute:latest “/bin/bash -c 'gunic…” 3 hours ago Up 3 hours 5005/tcp datagrok-grok_compute-1
6db4388f4384 postgres:12.9 “docker-entrypoint.s…” 3 hours ago Up 3 hours 0.0.0.0:5432->5432/tcp, :::5432->5432/tcp datagrok-db-1
fd25b9ef2845 datagrok/grok_connect:latest “java -Xmx4g -classp…” 3 hours ago Up 2 hours 0.0.0.0:1234->1234/tcp, :::1234->1234/tcp datagrok-grok_connect-1
7a4f1249bd49 datagrok/bio:2.7.2 “uvicorn --host 0.0.…” 9 days ago Exited (0) 3 hours ago datagrok_cvm_bio-2-7-2_1
263df9bd81bd hello-world “/hello” 6 weeks ago Exited (0) 6 weeks ago kind_euclid

Second one,
2023-07-21T10:42:31.502012484Z 10:42:31.497 [main] INFO grok_connect.GrokConnect - grok_connect - version: 2.1.3
2023-07-21T10:42:31.502111821Z 10:42:31.502 [main] INFO grok_connect.GrokConnect - Grok Connect initializing
2023-07-21T10:42:31.504690420Z 10:42:31.504 [main] INFO grok_connect.GrokConnect - Memory: free: 3799351808(99.52%), used: 18513408
2023-07-21T10:42:31.505056734Z 10:42:31.504 [main] DEBUG grok_connect.utils.ProviderManager - Initializing providers HashMap
2023-07-21T10:42:31.665140946Z 10:42:31.662 [main] INFO grok_connect.GrokConnect - grok_connect with Hikari pool
2023-07-21T10:42:31.665155278Z 10:42:31.663 [main] INFO grok_connect.GrokConnect - grok_connect: Running on http://localhost:1234
2023-07-21T10:42:31.665157477Z 10:42:31.663 [main] INFO grok_connect.GrokConnect - grok_connect: Connectors: [Cassandra, Access, Firebird, Hive2, Neptune, SAP HANA, Neo4j, Snowflake, SQLite, MySQL, Postgres, Oracle, Denodo, BigQuery, ClickHouse, DynamoDB, MariaDB, Impala, Teradata, Redshift, HBase, MS SQL, Hive, DB2, Vertica, PI, MongoDB, Athena, Virtuoso]
2023-07-21T10:54:22.746323866Z 10:54:22.744 [main] INFO grok_connect.GrokConnect - grok_connect - version: 2.1.3
2023-07-21T10:54:22.746332693Z 10:54:22.746 [main] INFO grok_connect.GrokConnect - Grok Connect initializing
2023-07-21T10:54:22.748684106Z 10:54:22.748 [main] INFO grok_connect.GrokConnect - Memory: free: 3799351808(99.52%), used: 18513408
2023-07-21T10:54:22.749036970Z 10:54:22.748 [main] DEBUG grok_connect.utils.ProviderManager - Initializing providers HashMap
2023-07-21T10:54:22.901728208Z 10:54:22.894 [main] INFO grok_connect.GrokConnect - grok_connect with Hikari pool
2023-07-21T10:54:22.901745507Z 10:54:22.894 [main] INFO grok_connect.GrokConnect - grok_connect: Running on http://localhost:1234
2023-07-21T10:54:22.901751055Z 10:54:22.894 [main] INFO grok_connect.GrokConnect - grok_connect: Connectors: [Cassandra, Access, Firebird, Hive2, Neptune, SAP HANA, Neo4j, Snowflake, SQLite, MySQL, Postgres, Oracle, Denodo, BigQuery, ClickHouse, DynamoDB, MariaDB, Impala, Teradata, Redshift, HBase, MS SQL, Hive, DB2, Vertica, PI, MongoDB, Athena, Virtuoso]
2023-07-21T10:59:30.311921681Z 10:59:30.310 [main] INFO grok_connect.GrokConnect - grok_connect - version: 2.1.3
2023-07-21T10:59:30.311937232Z 10:59:30.311 [main] INFO grok_connect.GrokConnect - Grok Connect initializing
2023-07-21T10:59:30.313580305Z 10:59:30.313 [main] INFO grok_connect.GrokConnect - Memory: free: 3799351808(99.52%), used: 18513408
2023-07-21T10:59:30.313870274Z 10:59:30.313 [main] DEBUG grok_connect.utils.ProviderManager - Initializing providers HashMap
2023-07-21T10:59:30.382954161Z 10:59:30.382 [main] INFO grok_connect.GrokConnect - grok_connect with Hikari pool
2023-07-21T10:59:30.383792854Z 10:59:30.382 [main] INFO grok_connect.GrokConnect - grok_connect: Running on http://localhost:1234
2023-07-21T10:59:30.383813931Z 10:59:30.383 [main] INFO grok_connect.GrokConnect - grok_connect: Connectors: [Cassandra, Access, Firebird, Hive2, Neptune, SAP HANA, Neo4j, Snowflake, SQLite, MySQL, Postgres, Oracle, Denodo, BigQuery, ClickHouse, DynamoDB, MariaDB, Impala, Teradata, Redshift, HBase, MS SQL, Hive, DB2, Vertica, PI, MongoDB, Athena, Virtuoso]
Thanks,
Taka

spodolskaya · July 21, 2023, 1:41pm

It seems that the service works properly. The issue should be connectivity between Datagrok and Grok Connect. To get more detailed information, could you please execute the following commands?

docker exec -it datagrok-datagrok-1 grep host /home/grok/cfg/prod/connectors.settings.json
docker exec -it datagrok-datagrok-1 curl grok_connect:1234/health

seritaka.gmail.com · July 21, 2023, 1:49pm

I could not get anything from these commands above.
Thanks,
Taka

spodolskaya · July 21, 2023, 1:52pm

Hmm, that seems odd. Perhaps we should use container IDs instead.

docker exec -it 524e3d4d14bf grep host /home/grok/cfg/prod/connectors.settings.json
docker exec -it 524e3d4d14bf curl grok_connect:1234/health

seritaka.gmail.com · July 22, 2023, 3:48am

I tried 1st command but could not get anything. So I logged in the container and checked the jsonfile.
{
“#type”: “ConnectorsServiceSettings”,
“externalStartup”: false,
“externalUse”: true,
“externalExe”: “”,
“externalHost”: “grok_connect”,
“externalPort”: 1234,
“externalDataFrameCompress”: true,
“sambaVersion”: “3.0”,
“sambaSpaceEscape”: “none”,
“queryMemoryLimitMb”: 2048,
“queryCellsLimit”: 50000000,
“debug”: false,
“connectionPoolTimerRate”: 60000,
“connectionPoolMaximumPoolSize”: 50,
“connectionPoolIdleTimeout”: 300000,
“localFileSystemAccess”: false,
“windowsSharesProxy”: “”
}

And I could not get response from curl command.
grok@524e3d4d14bf:~$ curl grok_connect:1234/health
grok@524e3d4d14bf:~$

Thanks

spodolskaya · July 24, 2023, 6:47am

Everything appears to be set up correctly.

Can you please attempt to execute the command curl -vkL grok_connect:1234/health within the datagrok container? This will allow us to gather additional details on why curl produced no output.

seritaka.gmail.com · July 24, 2023, 6:51am

Here is an output. I masked ip and port number of proxy.

$ curl -vkL grok_connect:1234/health

Uses proxy env variable no_proxy == ‘****’
Uses proxy env variable http_proxy == ‘*****’
Trying ******…
TCP_NODELAY set
Connected to ...* (...*) port **** (#0)

GET http://grok_connect:1234/health HTTP/1.1
Host: grok_connect:1234
User-Agent: curl/7.68.0
Accept: /
Proxy-Connection: Keep-Alive

Mark bundle as not supporting multiuse
< HTTP/1.1 503 Service Unavailable
< Content-Length: 0
< Connection: close
<
Closing connection 0

spodolskaya · July 24, 2023, 6:55am

Thank you for your prompt response. It appears that the problem is related to the Grok Connect application. We will conduct a more in-depth investigation on our end to determine why the container stopped responding correctly.

Could you kindly restart the grok_connect container, please?

seritaka.gmail.com · July 24, 2023, 6:57am

How can I restart the specific container？
Could you please give an example command to do it？
Thanks,

spodolskaya · July 24, 2023, 7:05am

Of course.
The easiest way is to delete the current container and start a new one. Follow these steps:

docker compose -f docker/localhost.docker-compose.yaml --project-name datagrok --profile grok_connect down
docker compose -f docker/localhost.docker-compose.yaml --project-name datagrok --profile grok_connect up -d

The location of localhost.docker-compose.yaml may be different on your computer.

seritaka.gmail.com · July 24, 2023, 8:53am

Thanks for your comment.
I tried it but grok_connect status is till failed.
Command is below.
$ docker compose -f localhost.docker-compose.yaml --project-name datagrok --profile grok_connect down
[+] Running 1/0
! Network datagrok_datagrok Resource is still in use 0.0s
$ docker compose -f localhost.docker-compose.yaml --project-name datagrok --profile grok_connect up -d
[+] Building 0.0s (0/0)
[+] Running 1/1
Container datagrok-grok_connect-1 Started

spodolskaya · July 24, 2023, 9:44am

I will conduct further investigation. Thank you for providing the debug information. I will provide you with a solution by the end of the day.

seritaka.gmail.com · January 30, 2024, 7:54am

Hi, I could fix the issue with following script!
#–remove-orphane option is required to down the container.

$ docker compose -f docker/localhost.docker-compose.yaml --project-name datagrok --profile grok_connect down --remove-orphans

$ docker compose -f docker/localhost.docker-compose.yaml --project-name datagrok --profile grok_connect up -d

Then restart datagrok, after that all systems seem work fine
Thanks!