Published November 2023
Apache Pulsar servers may need occasional restarts to maintain optimal performance and keep current with software updates. Thoughtful planning is needed to perform these restarts correctly and avoid disruption. In this instructional guide, we will walk you through the best practices for restarting a Pulsar server.
Reasons for Restarting a Pulsar Server
The most common reasons for restarting a Pulsar server are because of config changes, software updates, and performance issues.
With configuration changes, a restart might be necessary when changing topic or security settings. Additionally, a restart is often required to apply software updates.
Performance issues and troubleshooting are also common reasons for restarting Pulsar servers. Pulsar servers may accumulate resource usage or experience performance issues over time. Restarting the Pulsar server can clear resource bottlenecks and improve performance.
Finally, restarting your Pulsar service can aid troubleshooting and diagnostics.
Preparation for Restarting Pulsar
Proper preparation will ensure a smooth restart. Firstly and most importantly, back up your Pulsar data before beginning your restart. Backups are a critical step because data loss during a server restart can be problematic.
Learn more about data backups here.
Next, notify your team and any other users about the planned server restart. Coordinate with stakeholders for minimal disruption to data processing and applications.
You will also want to identify any dependencies. This can include consumers, producers, or external systems. Ensure that they are also prepared for the Pulsar server restart and can handle potential downtime.
Thoroughly check any configuration changes you plan to apply during the restart. Ensure that they are tested in a non-production environment before applying them to your live Pulsar cluster.
Learn how to build a Pulsar testing environment.
Lastly, use Pulsar monitoring tools to assess the current state of your Pulsar cluster. Monitoring will identify any existing issues or unusual behavior before the restart. OpenSearch and OpenSearch Dashboards are great tools for monitoring Pulsar.
Instructions for Restarting a Pulsar Server
Use the following command to stop the Pulsar broker:
bin/Pulsar-server-stop.sh
This command allows Pulsar to finish processing any pending messages and release resources appropriately.
After issuing the shutdown command, monitor the broker’s logs to check that it has shut down correctly. You should see log messages indicating that the broker has stopped.
Now is the time to apply configuration changes or updates. Make the necessary changes to your configuration files and restart the Pulsar broker using the following command:
bin/Pulsar-server-start.sh config/server.properties
This will start the Pulsar broker with the updated configuration.
As the Pulsar broker restarts, closely monitor its logs and metrics to confirm that it comes back online smoothly. Pay attention to any error messages or warnings that indicate issues during the restart.
Now that Pulsar is back online, check that data replication is working (if applicable). Confirm that the leaders and followers are in sync.
Test the connectivity of your Pulsar cluster by producing and consuming test messages. Verify that your applications and clients can connect to the cluster and function as expected.
Post-Restart Validation
A post-restart validation will confirm that everything is functioning as expected.
You’ll first want to check that your data is identical to the backup you created before the restart. Check that no data loss or corruption occurred during the restart process.
Review the Pulsar broker logs for any warnings or errors that occurred during the restart. If there are any issues, then investigate and address them.
Continue to monitor your Pulsar cluster and address any performance issues that arose during the restart.
Summing it up
While restarting an Apache Pulsar server is a routine maintenance task, it still requires careful planning.
Remember that each Pulsar cluster is unique. Always adapt these best practices to suit your specific environment and requirements.
Published by