Errors when running ansible-playbook certbot.yml

[operations]

Please Help. Thanks in Advance.

Please note: we are using the GitHub - primeroIMS/primero: Primero is an application designed to help child protection workers and social workers in humanitarian and development contexts manage data on vulnerable children and survivors of violence. Please carefully read our LICENSE. If you would like access to the CPIMS+ and GBVIMS+ configurations, please contact: childprotectioninnovation@gmail.com [master branch]

Our DNS records are pointing to this Ubuntu server public IP. And if we install a webserver on the host itself, we were able to reach and was able to run lets encrypt to make it https. But for Primero communication to the docker instance is not happening. And we are getting Connection refused error.

When running the last command ansible-playbook certbot.yml.

Following documentation was referred for doing the install on the Ubuntu server 20.04

primeroIMS/primero/blob/master/ansible/README.md

Self Hosting Primero v2 - Installation

So maybe you have read the programmatic guidance for self hosting, self hosting set up and background info, the hosting requirements checklist, you have completed all the security tasks, you have reviewed the Tier 4 Deployment Workplan, you have hired world class dev ops engineers/system admins and topped your bank account up to overflowing. Looks like you may be ready to self host! Target Install Ubuntu 20.04 LTS and upgrade the target machine if you have not already done so: sudo apt-get …

We are getting following errors:

PLAY [protect.kinshipunited.org] *****************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************************************************************************************************************************
ok: [protect.kinshipunited.org]

TASK [primero-certbot : create the /srv/primero/bin directory] ***********************************************************************************************************************************************************************
ok: [protect.kinshipunited.org]

TASK [primero-certbot : copy the certbot binary to /srv/primero/bin/certbot] *********************************************************************************************************************************************************
ok: [protect.kinshipunited.org]

TASK [primero-certbot : create the /srv/primero/bin/run-certbot.sh script] ***********************************************************************************************************************************************************
ok: [protect.kinshipunited.org]

TASK [primero-certbot : run certbot] *************************************************************************************************************************************************************************************************
fatal: [protect.kinshipunited.org]: FAILED! => changed=true
cmd:

• /srv/primero/bin/run-certbot.sh
  delta: ‘0:00:04.039422’
  end: ‘2023-03-01 21:15:07.741754’
  msg: non-zero return code
  rc: 1
  start: ‘2023-03-01 21:15:03.702332’
  stderr: |-
  • exec /srv/primero/bin/certbot -d protect.kinshipunited.org --cert-name primero -m @.org -p primero
    Saving debug log to /var/log/letsencrypt/letsencrypt.log
    Plugins selected: Authenticator webroot, Installer None
    Performing the following challenges:
    http-01 challenge for protect.kinshipunited.org
    Using the webroot path /usr/share/nginx/html for all unmatched domains.
    Waiting for verification…
    Challenge failed for domain protect.kinshipunited.org
    http-01 challenge for protect.kinshipunited.org
    Cleaning up challenges
    Some challenges have failed.
    Traceback (most recent call last):
    File “/srv/primero/bin/certbot”, line 141, in
    code = _main(sys.argv)
    File “/srv/primero/bin/certbot”, line 128, in _main
    _check_call(command)
    File “/srv/primero/bin/certbot”, line 26, in _check_call
    subprocess.check_call(args, *popenargs, **kwargs)
    File “/usr/lib/python3.8/subprocess.py”, line 364, in check_call
    raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command ‘[‘docker’, ‘run’, ‘–rm’, ‘–interactive’, ‘–volume’, ‘primero_certbot_certificates:/etc/letsencrypt’, ‘–volume’, ‘primero_certbot_challenges:/usr/share/nginx/html/.well-known/acme-challenge’, ‘–volume’, ‘/tmp/tmpruuwzzdp:/tmp/deploy-hook-dir’, ‘certbot/certbot:v1.11.0’, ‘certonly’, ‘–non-interactive’, ‘–domain’, ‘protect.kinshipunited.org’, ‘–cert-name’, ‘primero’, ‘–email’, ‘@.org’, ‘–agree-tos’, ‘–rsa-key-size’, ‘2048’, ‘–deploy-hook’, ‘touch /tmp/deploy-hook-dir/renewed && chmod a+r -R /etc/letsencrypt/live && chmod a+r -R /etc/letsencrypt/archive’, ‘–webroot’, ‘–webroot-path’, ‘/usr/share/nginx/html’]’ returned non-zero exit status 1.
    stderr_lines:
    stdout: |-
    Requesting a certificate for protect.kinshipunited.org
    IMPORTANT NOTES:

  • The following errors were reported by the server:

    Domain: protect.kinshipunited.org
    Type: connection
    Detail: 44.213.232.134: Fetching
    http://protect.kinshipunited.org/.well-known/acme-challenge/jKdGBJqIKQcvT6KHg-qs9VMMg3_B6ytfP1ipKWAwT1E:
    Error getting validation data

    To fix these errors, please make sure that your domain name was
    entered correctly and the DNS A/AAAA record(s) for that domain
    contain(s) the right IP address. Additionally, please check that
    your computer has a publicly routable IP address and that no
    firewalls are preventing the server from communicating with the
    client. If you’re using the webroot plugin, you should also verify
    that you are serving files from the webroot path you provided.

  • docker run --rm --interactive --volume primero_certbot_certificates:/etc/letsencrypt --volume primero_certbot_challenges:/usr/share/nginx/html/.well-known/acme-challenge --volume /tmp/tmpruuwzzdp:/tmp/deploy-hook-dir certbot/certbot:v1.11.0 certonly --non-interactive --domain protect.kinshipunited.org --cert-name primero --email @.org --agree-tos --rsa-key-size 2048 --deploy-hook ‘touch /tmp/deploy-hook-dir/renewed && chmod a+r -R /etc/letsencrypt/live && chmod a+r -R /etc/letsencrypt/archive’ --webroot --webroot-path /usr/share/nginx/html
    stdout_lines:

PLAY RECAP ***************************************************************************************************************************************************************************************************************************
protect.kinshipunited.org : ok=4 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

providing config details:

/primero/ansible/inventory/inventory.yml

all:
hosts:
protect.kinshipunited.org:
ansible_user: ‘kinship’
primero_host: ‘protect.kinshipunited.org’
primero_tag: ‘latest’

As of Primero v2.5, the default Docker deployed PostgreSQL is 10.22.

If you want to run Primero with a different version of PostgreSQL,

set primero_postgres_version to either ‘11’ or ‘14’.

NOTE: YOU NEED TO PERFORM A DATA MIGRATION BEFORE CHANGING POSTGRES VERSIONS!!!

primero_postgres_version: ‘14’

locale_all: ‘en,fr,ar’
always_pull: true,

These 3 variables are used to drive the build task.

Can be omitted if pulling images from Dockerhub

primero_repo_branch: ‘master’
build_docker_tag: ‘latest’
build_docker_container_registry: ‘’

If you want to seed from a private configuration repo

primero_configuration_repo: ‘git@bitbucket.org:quoin/primero-x-configuration.git’

primero_configuration_repo_branch: ‘master’
primero_configuration_path: ‘directory/of/config/loader/script’
use_lets_encrypt: ‘true’
use_external_certs: ‘false’

The other certbot/lets_encrypt variables are optional. Include only if using Let’s Encrypt

certbot_domain:

• ‘{{ primero_host }}’
  certbot_email: ‘removed@actual-domain-removed’
  lets_encrypt_domain: ‘{{ primero_host }}’
  lets_encrypt_email: ‘{{ certbot_email }}’

Different for self-signed certs

nginx_ssl_cert_path: ‘/etc/letsencrypt/live/primero/fullchain.pem’
nginx_ssl_key_path: ‘/etc/letsencrypt/live/primero/privkey.pem’

nginx_ssl_cert_path: ‘/home/kinship/my-external-certs/kust.crt’

nginx_ssl_key_path: ‘/home/kinship/my-external-certs/kupr.key’

Optionally parametrize Primero containers with other environment attributes.

Do not use this dictionary for secrets!!!

environment_variables:
PRIMERO_DEFAULT_USERS: ‘true’ # Seed default users when using the managed Primero SaaS configs

LOCALE_DEFAULT: ‘ar’ # Optionally override English as the default locale.

---

/primero/ansible/secrets.yml
primero_secret_key_base: ‘---------dummy-value01-----------’
primero_message_secret: ‘---------dummy-value02-----------’
postgres_password: ‘---------dummy-value03-----------’
devise_secret_key: ‘---------dummy-value04-----------’
devise_jwt_secret_key: ‘---------dummy-value05-----------’

[irlawrence]

You have removed the webserver you installed on the host correct?

[operations]

Yes.

In the second scenario: We modified the inventory file to refer to external certs (with our own certificate and key), instead of letsencrypt; but it doesn’t seem to work either.

Is there any other documentation that I can refer that supports Ubuntu 20.04 version. Other than the one which I was looking at:

I’m ready to discard the instance that we have so far and will start over; based on your suggestions. Thank you.

primeroIMS/primero/blob/master/ansible/README.md

Self Hosting Primero v2 - Installation

So maybe you have read the programmatic guidance for self hosting, self hosting set up and background info, the hosting requirements checklist, you have completed all the security tasks, you have reviewed the Tier 4 Deployment Workplan, you have hired world class dev ops engineers/system admins and topped your bank account up to overflowing. Looks like you may be ready to self host! Target Install Ubuntu 20.04 LTS and upgrade the target machine if you have not already done so: sudo apt-get …

[operations]

Some more information:

But here are the problems that we have. When letsencrypt is trying to validate the container through the DNS. It cannot even reach the container and it’s contents.

For example: Following file/directory in DOCKER doesn’t even exist
/.well-known/acme-challenge/jKdGBJqIKQcvT6KHg-qs9VMMg3_B6ytfP1ipKWAwT1E

And when I connected to the container and created a simple file; that file cannot be accessed as well. The error message that I’m getting from the browser is “ERR_CONNECTION_REFUSED”

Used this to connect to the NGINX container; which was created as part of ansible:
sudo -s
docker ps
docker exec -it sh

[irlawrence]

It looks like something may be messed up with ther nginx container

[operations]

That’s right. Certbot is trying to validate with nginx container with following location.
/usr/share/nginx/html/.well-known/acme-challenge

Can you help me understand where from “acme” or “acme-challenge” is coming from?

[irlawrence]

@operations

acme-challenge

It comes from Lets Encrypt when you run it

[operations]

Hello,

What is the default user id and password to use after getting the login page? Following username and password did not work.
UserName: primero
Password: primer0!

Please note that deployment was done with ansible.

Thank you.

[irlawrence]

You can create one in the application container using something like:

docker exec -it <container_name> -- bash

then

rails -c
unicef = Agency.find_by(agency_code: 'UNICEF')
role = Role.find_by_name('Superuser')

User.create_or_update!({
'user_name' => 'a@b.com',
'password' => 'whatever_you_want',
'password_confirmation' => 'whatever_you_want',
'full_name' => 'A Name',
'email' => 'a@b.com',
'disabled' => 'false',
'agency_id' => unicef.id,
'role_id' => role.id,
'user_groups' => UserGroup.all,
'locale' => Primero::Application::LOCALE_ENGLISH
	})

Which adds an English speaking superuser into all groups in the UNICEF agency

[operations]

When I connect to the docker container and run “railc -c”; I’m getting following error. Please suggest.

docker exec -it 1ab4d9e812f9 sh

/srv/primero/application $ rails -c
Rspec not installed–omitting tasks
rails aborted!
Don’t know how to build task ‘default’ (See the list of available tasks with rails --tasks)

(See full trace by running task with --trace).

Some additional information::

/srv/primero/application $ rails about
Rspec not installed–omitting tasks
D, [2023-03-21T18:09:10.315883 #63] DEBUG – : (0.3ms) SELECT 1;
I, [2023-03-21T18:09:10.317677 #63] INFO – : Setting up PeriodicJobs
D, [2023-03-21T18:09:10.337422 #63] DEBUG – : Delayed::Backend::ActiveRecord::Job Exists? (0.7ms) SELECT 1 AS one FROM “delayed_jobs” WHERE (handler LIKE ‘%job_class: ArchiveBulkExports%’) LIMIT $1 [[“LIMIT”, 1]]
D, [2023-03-21T18:09:10.341865 #63] DEBUG – : Delayed::Backend::ActiveRecord::Job Exists? (0.6ms) SELECT 1 AS one FROM “delayed_jobs” WHERE (handler LIKE ‘%job_class: OptimizeSolr%’) LIMIT $1 [[“LIMIT”, 1]]
D, [2023-03-21T18:09:10.343499 #63] DEBUG – : Delayed::Backend::ActiveRecord::Job Exists? (0.6ms) SELECT 1 AS one FROM “delayed_jobs” WHERE (handler LIKE ‘%job_class: RecalculateAge%’) LIMIT $1 [[“LIMIT”, 1]]
D, [2023-03-21T18:09:10.802067 #63] DEBUG – : (1.3ms) SELECT “schema_migrations”.“version” FROM “schema_migrations” ORDER BY “schema_migrations”.“version” ASC
About your application’s environment
Rails version 6.1.7.1
Ruby version ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x86_64-linux-musl]
RubyGems version 3.1.6
Rack version 2.2.6.2
Middleware ActionDispatch::HostAuthorization, ActionDispatch::SSL, Rack::Sendfile, ActionDispatch::Executor, Rack::Runtime, Rack::MethodOverride, ActionDispatch::RequestId, ActionDispatch::RemoteIp, Rails::Rack::Logger, ActionDispatch::ShowExceptions, ActionDispatch::DebugExceptions, ActionDispatch::ActionableExceptions, ActionDispatch::Callbacks, ActionDispatch::Cookies, ActionDispatch::ContentSecurityPolicy::Middleware, ActionDispatch::PermissionsPolicy::Middleware, Rack::Head, Rack::ConditionalGet, Rack::ETag, Rack::TempfileReaper, WwwAuthenticate, Warden::Manager, Rack::Deflater, JwtTokenSetter, Warden::JWTAuth::Middleware, Rack::Attack
Application root /srv/primero/application
Environment production
Database adapter postgresql
Database schema version 20220328075916

[irlawrence]

There is a difference between doing

docker exec -it 1ab4d9e812f9 sh
rails c

and doing this (which should work)

docker exec -it primero_application_1 bash
rails c

[operations]

Thank you. I was able to connect and run the commands you gave. But getting following error message “NoMethodError (undefined method `id’ for nil:NilClass)”

irb(main):001:0> unicef=Agency.find_by(agency_code: ‘UNICEF’)
D, [2023-03-21T19:32:06.354513 #119] DEBUG – : Agency Load (1.3ms) SELECT “agencies”.* FROM “agencies” WHERE “agencies”.“agency_code” = $1 LIMIT $2 [[“agency_code”, “UNICEF”], [“LIMIT”, 1]]
=> nil
irb(main):002:0> role=Role.find_by_name(‘Superuser’)
D, [2023-03-21T19:32:07.338724 #119] DEBUG – : Role Load (1.0ms) SELECT “roles”.* FROM “roles” WHERE “roles”.“name” = $1 LIMIT $2 [[“name”, “Superuser”], [“LIMIT”, 1]]
=> nil
irb(main):003:0> User.c’password_confirmation’ => ‘temp-password-01’,
‘full_name’ => ‘kinship admin’,
‘email’ => ‘email@temp-email.org’,
‘disabled’ => ‘false’,
‘agency_id’ => unicef.id,
‘role_id’ => role.id,
‘user_groups’ => UserGroup.all,
‘locale’ => Primero::Application::LOCALE_ENGLISH
irb(main):003:2* User.create_or_update!({
irb(main):004:2* ‘user_name’ => ‘operations’,
irb(main):005:2* ‘password’ => ‘temp-password-01’,
irb(main):006:2* ‘password_confirmation’ => ‘temp-password-01’,
irb(main):007:2* ‘full_name’ => ‘kinship admin’,
irb(main):008:2* ‘email’ => ‘email@temp-email.org’,
irb(main):009:2* ‘disabled’ => ‘false’,
irb(main):010:2* ‘agency_id’ => unicef.id,
irb(main):011:2* ‘role_id’ => role.id,
irb(main):012:2* ‘user_groups’ => UserGroup.all,
irb(main):013:2* ‘locale’ => Primero::Application::LOCALE_ENGLISH
irb(main):014:0> })
Traceback (most recent call last):
1: from (irb):10
NoMethodError (undefined method `id’ for nil:NilClass)

irb(main):015:0>

[operations]

Since we are initially running unicef= and role= with some values. I changed User.create_or_update! to reflect with those specific names instead of .id. Now I’m getting this error::
NameError (undefined local variable or method `userGroup’ for main:Object)

::::: details below ::::

irb(main):004:2* User.create_or_update!({
irb(main):005:2* ‘user_name’ => ‘operations’,
irb(main):006:2* ‘password’ => ‘temp-password-01’,
irb(main):007:2* ‘password_confirmation’ => ‘temp-password-01’,
irb(main):008:2* ‘full_name’ => ‘kinship admin’,
irb(main):009:2* ‘email’ => ‘email@temp-email.org’,
irb(main):010:2* ‘disabled’ => ‘false’,
irb(main):011:2* ‘agency_id’ => unicef,
irb(main):012:2* ‘role_id’ => role,
irb(main):013:2* ‘user_groups’ => userGroup.all,
irb(main):014:2* ‘locale’ => Primero::Application::LOCALE_ENGLISH
irb(main):015:0> })
Traceback (most recent call last):
1: from (irb):12
NameError (undefined local variable or method `userGroup’ for main:Object)
irb(main):016:0>

[irlawrence]

So hard coding things is not going to work as the issue is deeper. You need to find out why the command you ran to SELECT all agencies on your system returned nil

@operations

irb(main):001:0> unicef=Agency.find_by(agency_code: ‘UNICEF’)
D, [2023-03-21T19:32:06.354513 #119] DEBUG – : Agency Load (1.3ms) SELECT “agencies”.* FROM “agencies” WHERE “agencies”.“agency_code” = $1 LIMIT $2 [[“agency_code”, “UNICEF”], [“LIMIT”, 1]]
=> nil

on my test system it returns this →

SELECT  "agencies".* FROM "agencies" WHERE "agencies"."agency_code" = $1 LIMIT $2  [["agency_code", "UNICEF"], ["LIMIT", 1]]
=> #<Agency id: 1, unique_id: "UNICEF", agency_code: "UNICEF", order: 0, name_i18n: {"en"=>"UNICEF", "pt-BR"=>"UNICEF"}, description_i18n: nil, telephone: nil, services: [], logo_enabled: false, disabled: false, pdf_logo_option: false, exclude_agency_from_lookups: false, terms_of_use_enabled: false, created_at: "2021-09-27 21:07:10", updated_at: "2021-09-27 21:39:41", logo_full_base64: nil, logo_full_file_name: nil, logo_icon_base64: nil, logo_icon_file_name: nil, terms_of_use_base64: nil, terms_of_use_file_name: nil>

so basically the database looks empty. Did the seed finish correctly?

[irlawrence]

You can check the database by exec’ing into the postgres container and running psql

docker exec -it primero_postgres_1 bash
bash-4.4# psql -U primero

[operations]

From the “primero_production” database I was able to see tables; and within the agencies table, columns are also shown.
You are right, I’m not seeing any data within the columns.

I actually rerun all the ansible scripts again; and did not receive any errors. And still don’t see data.

Since the docker is running now. Is there a way that I can run the seed within the docker container?

Thanks in advance.

[operations]

Hi irlawrence,

Thank you for the suggestion. I connected to the container and ran these commands; and it created the data. And I was able to login to the application now.

docker exec -it primero_application_1 bash
rails c

rails db:create
rails db:migrate
rails db:seed

Thank you so much.

[irlawrence]

Awesome! Hope you enjoy it

[srinivasa.puneeth]

Hey @operations, I am facing the same error when running “ansible-playbook certbot.yml”. I went through the whole conversation here on this topic but I did not find a solution on how did you resolve this issue.

we are using the: https://github.com/primeroIMS/primero [developementV2 branch]

DNS records are pointing to our Ubuntu public IP and I am getting the same error as you did when I am running the last command from the self-hosting V2 Primero documentation: ansible-playbook certbot.yml.

Can you or @irlawrence help me out here? Thanks in advance.