fbpx

Day: December 14, 2022

Docker Compose For Elasticserach with Persistent Volume

by

To create a Docker Compose file for Elasticsearch with a persistent volume, you can use the volumes and volume_driver options in the elasticsearch service configuration. Here is an example docker-compose.yml file:​

				
					version: '3.7'

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.9.1
    volumes:
      - data:/usr/share/elasticsearch/data
    environment:
      - node.name=elasticsearch
      - cluster.name=elasticsearch
      - discovery.seed_hosts=elasticsearch
      - cluster.initial_master_nodes=elasticsearch
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    ports:
      - 9200:9200
    networks:
      - elasticsearch

volumes:
  data:
    driver: local
    driver_opts:
      o: bind
      type: none
      device: /path/to/data/directory

networks:
  elasticsearch:

This docker-compose.yml file will create a Docker container for Elasticsearch, and mount a persistent volume at the path /path/to/data/directory on the host machine. This volume will be used to store the data for Elasticsearch.

To use this docker-compose.yml file, save it to a file on your local machine, and run the following command from the directory where the file is saved:

				
					docker-compose up -d
docker-compose ps

This will start the Elasticsearch container and mount the persistent volume.

I hope this helps! Let me know if you have any other questions.

Backup Elasticserach From shell script

by

To backup Elasticsearch from a shell script, you can use the Elasticsearch API and a tool like curl to make a request to create a snapshot of your Elasticsearch cluster.

Here’s an example of how you might do this:

				
					#!/bin/bash

# Set the URL for the Elasticsearch snapshot API
SNAPSHOT_URL="http://localhost:9200/_snapshot/my_backup_repository"

# Set the name for the snapshot
SNAPSHOT_NAME="my_backup"

# Use curl to create the snapshot
curl -XPUT "$SNAPSHOT_URL/$SNAPSHOT_NAME"

This script will create a snapshot of your Elasticsearch cluster and store it in the repository named “my_backup_repository”. The snapshot will be named “my_backup”.

You can then restore this snapshot at a later time if necessary. To do this, you can use the Elasticsearch API and curl again, like this

				
					# Set the URL for the Elasticsearch restore API
RESTORE_URL="http://localhost:9200/_snapshot/my_backup_repository/my_backup/_restore"

# Use curl to restore the snapshot
curl -XPOST "$RESTORE_URL"

# Set the URL for the Elasticsearch restore API
RESTORE_URL=”http://localhost:9200/_snapshot/my_backup_repository/my_backup/_restore”

# Use curl to restore the snapshot
curl -XPOST “$RESTORE_URL”

Insert JSON Data File into PostgreSQL using Python Script

by

the psycopg2 library to connect to the database and execute an INSERT statement. Here is an example:

				
					import json
import psycopg2

# read JSON data from file
with open('data.json') as f:
    data = json.load(f)

# connect to the database
conn = psycopg2.connect(host="localhost", database="mydb", user="user", password="pass")

# create a cursor
cur = conn.cursor()

# execute the INSERT statement
cur.execute("INSERT INTO table_name (name, email, city) VALUES (%s, %s, %s)", (data["name"], data["email"], data["city"]))

# commit the changes to the database
conn.commit()

# close the cursor and connection
cur.close()
conn.close()

This code reads the JSON data from a file named “data.json”, and then uses the psycopg2 library to connect to a Postgres database. A cursor is used to execute an INSERT statement with the JSON data as the values. The changes are then committed to the database and the cursor and connection are closed.

Note that this is just an example, and you will need to modify the code to match the specific structure of your database and JSON data. You may also want to add error handling and additional functionality as needed. For example, you could read multiple JSON files or insert multiple records in a loop.

Create Insert SQL File with 100 Records From Python faker

by

To create an INSERT SQL file with 100 records using the Faker library in Python, you can use the following code:

				
					from faker import Faker

fake = Faker()

# create list to store SQL statements
sqls = []

# generate fake data and create INSERT SQL statements
for i in range(100):
    name = fake.name()
    email = fake.email()
    city = fake.city()
    sql = "INSERT INTO table_name (name, email, city) VALUES ('{}', '{}', '{}')".format(name, email, city)
    sqls.append(sql)

# write SQL statements to file in /tmp directory
with open(r'C:\Users\Administrator\Downloads\fold\insert.sql', 'w') as f:
    for i, sql in enumerate(sqls):
        f.write(sql + '\n')
        print('Inserted record {} of 100'.format(i + 1))

This code generates 100 fake records for a name, email, and city using the Faker class, and then creates an INSERT SQL statement for each record. The SQL statements are stored in a list, and then written to a file named “insert.sql”, with each statement on a separate line.

Note that this is just an example, and you will need to modify the code to match the specific structure of your database table. You may also want to add error handling and additional functionality as needed. For example, you could generate more complex SQL statements with multiple values or different data types.

Create Insert SQL File with Python Facker

by

To create an INSERT SQL file using the Faker library in Python, you can use the following code:

This code generates fake data for a name, email, and city using the Faker class, and then creates an INSERT SQL statement using the fake data. The SQL statement is then written to a file named “insert.sql”.

Note that this is just an example, and you will need to modify the code to match the specific structure of your database table. You may also want to add error handling and additional functionality as needed. For example, you could generate multiple fake records and write them to the file in a loop, or you could generate more complex SQL statements with multiple values or different data types.

				
					from faker import Faker

fake = Faker()

# generate fake data
name = fake.name()
email = fake.email()
city = fake.city()

# create INSERT SQL statement
sql = "INSERT INTO table_name (name, email, city) VALUES ('{}', '{}', '{}')".format(name, email, city)

# write SQL statement to file
with open('insert.sql', 'w') as f:
    f.write(sql)

Python Script to Copy only Changed Files From one Dir to Another

by

Here is a python script that will copy only changed files from one directory to another:

				
					import os
import shutil

# source directory
src = r"C:\Users\Administrator\Downloads\fold"

# destination directory
dst = r"C:\Users\Administrator\Downloads\foldc"

# copy changed files
for item in os.listdir(src):
    s = os.path.join(src, item)
    d = os.path.join(dst, item)
    if os.path.isdir(s):
        continue
    if not os.path.exists(d) or os.stat(s).st_mtime - os.stat(d).st_mtime > 1:
        shutil.copy2(s, d)

This script will copy files from the source directory to the destination directory if they do not already exist in the destination directory, or if they have been modified more recently than the version in the destination directory. It uses the shutil module’s copy2() function to copy the files.

Note that this script will not copy directories, and it will not handle errors, such as if the destination directory does not exist or if the user does not have permission to write to the destination directory. You may want to add error handling and additional functionality to the script to handle these cases.

Script to copy both Files and Folders

				
					import os
import shutil

# source directory
src = r'C:\Users\Administrator\Downloads\fold'

# destination directory
dst = r'C:\Users\Administrator\Downloads\foldc'

# copy changed files and directories
for item in os.listdir(src):
    s = os.path.join(src, item)
    d = os.path.join(dst, item)
    if os.path.isdir(s):
        if not os.path.exists(d):
            shutil.copytree(s, d)
        else:
            for f in os.listdir(s):
                sf = os.path.join(s, f)
                df = os.path.join(d, f)
                if os.path.isdir(sf):
                    continue
                if not os.path.exists(df) or os.stat(sf).st_mtime - os.stat(df).st_mtime > 1:
                    shutil.copy2(sf, df)
    else:
        if not os.path.exists(d) or os.stat(s).st_mtime - os.stat(d).st_mtime > 1:
            shutil.copy2(s, d)