Skip to content

Conversation

@CarolineDenis
Copy link
Contributor

@CarolineDenis CarolineDenis commented Jun 16, 2025

Fixes #2931
Fixes #4832
Fixes #6210

Adds an initial set up page for configuring your database much like the Spwizard for Specify 6.

image

TODO:

  • Title colors are white-on-white.
  • Make certain fields required (at least for user)
  • Error on backend doesn't stop you from progressing onto the next step.
  • Defaults for fields (So certain tree ranks can be true by default)
  • Progress indicator?
  • Backend should tell frontend which step to go to next. Needed to conditionally skip geography tree setup pages.
  • Setup can be finished before schema is finished being created. Need to have some sort of waiting page, or speed improvement? Thinking I can change the setup so you fill out the forms first, then at the very end the database is created.
  • Use new default files Feat (setup tool): Create default files for the new setup tool #7444
    • Schema overrides
    • Schema localization (english only)
    • Picklists and Global Picklists
    • Preptypes
    • Taxon treedef (per discipline)
    • Geography treedef
    • Storage treedef
  • Create trees
  • Add documentation links
  • Only use localized strings
  • Add better error handling. User should be notified and sent back to the forms if anything goes wrong.
  • Check if worker is set up before continuing.

Future TODO:

  • Password confirmation input doesn't work if you input the password into it first, then fill out the normal password field.
  • Notification polling seems to be causing errors on the backend when lingering on the setup page. Will investigate.
  • Need schema for API
  • Save form to localStorage. Also change the Save & Continue text maybe.
  • Normalize all keys (recursively) on backend api requests
  • Institution should have a value for all treedef fields once all trees are created. There should be api.finalize_institution or something similar.
  • There should be a functioning progress bar during the setup.
  • Speed up schema config localization setup. This is the biggest bottleneck in the setup process and it can be sped up significantly.
  • Fix two geography tree forms showing up on the overview sidebar.
  • Add documentation links and descriptions to all resources

Checklist

  • Self-review the PR after opening it to make sure the changes look good and
    self-explanatory (or properly documented)
  • Add relevant issue to release milestone
  • Add pr to documentation list
  • Add automated tests
  • Add a reverse migration if a migration is present in the PR

Testing instructions

  • Use a blank DB to test this PR.
  • Fill out the setup forms
    • Make sure the forms look good in light mode and dark mode (It uses your system settings.)
    • You can only progress to the next form if you filled out all required fields.
    • Make sure your are shown in the Overview sidebar.
    • Make sure you can submit at the end.
  • Make sure you can log into the database.
  • Make sure the schema config defaults were applied correctly.
  • Make sure default picklists were created correctly.
  • Make sure prep types were created correctly.
  • Make sure all tree viewer pages load.
  • Make sure API endpoints work
    • setup_tool/setup_database/create
    • setup_tool/institution/create
    • setup_tool/division/create
    • setup_tool/discipline/create
    • setup_tool/collection/create
    • setup_tool/specifyuser/create
  • Local testing:
    • Use an empty DB again (restore it or use a different one).
    • Stop the specify worker and try to start the setup. You should get a clear error on the frontend.

@grantfitzsimmons
Copy link
Member

Any specifyuser can be set as an institution admin by inserting the following, replacing the specifyuser_id with the ID of the newly created user:

INSERT INTO spuserpolicy (id, resource, action, collection_id, specifyuser_id) VALUES (1, '%', '%', null, 1);

@CarolineDenis
Copy link
Contributor Author

CarolineDenis commented Jul 9, 2025

NOTES:

@alesan99
Copy link
Contributor

Hey @specify/dev-testing, I'm requesting an early review for this PR since I'm pretty close to being done (with the most important requirements at least).

I have my TODO list in the PR description, I set aside the more minor issues to be addressed in separate issues.

I'd appreciate any feedback or suggestions on the UI, code, and general setup process

image

@alesan99 alesan99 requested review from a team and grantfitzsimmons October 30, 2025 15:25
@alesan99 alesan99 marked this pull request as ready for review October 30, 2025 15:25
@melton-jason melton-jason self-requested a review October 30, 2025 15:27
Copy link
Member

@grantfitzsimmons grantfitzsimmons left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using a brand new database (never before created), it fails to start and get running.

2025-10-31 15:32:54 Updating static files in /volumes/static-files/.
2025-10-31 15:32:56 Applying Django migrations.
2025-10-31 15:32:56 Starting MariaDB database and user creation script...
2025-10-31 15:32:56 --------------------------------------------------
2025-10-31 15:32:56 DB Configuration:
2025-10-31 15:32:56   DB Host: host.docker.internal
2025-10-31 15:32:56   DB Port: 3306
2025-10-31 15:32:56   DB Name: somethingtotallynew
2025-10-31 15:32:56   Master User: root
2025-10-31 15:32:56   Migrator User: specify_migrator
2025-10-31 15:32:56   App User: specify_user
2025-10-31 15:32:56 --------------------------------------------------
2025-10-31 15:32:56 Checking if MariaDB instance is up and running...
2025-10-31 15:32:56 MariaDB is up and running.
2025-10-31 15:33:02 Creating database 'somethingtotallynew'...
2025-10-31 15:33:02 Executing: mysql -h "host.docker.internal" -P "3306" -u "root" --password="<hidden>" -e "CREATE DATABASE `somethingtotallynew`;"
2025-10-31 15:33:03 Creating user 'specify_migrator'...
2025-10-31 15:33:03 Executing: mysql -h "host.docker.internal" -P "3306" -u "root" --password="<hidden>" -e "CREATE USER 'specify_migrator'@'%' IDENTIFIED BY '<hidden>';"
2025-10-31 15:33:07 Granting privileges to new user...
2025-10-31 15:33:07 Executing: mysql -h "host.docker.internal" -P "3306" -u "root" --password="<hidden>" -e "GRANT ALL PRIVILEGES ON `somethingtotallynew`.* TO 'specify_migrator'@'%'; FLUSH PRIVILEGES;"
2025-10-31 15:33:08 Notice: 'specify_migrator'@'%' lacks usable access to 'somethingtotallynew'.
2025-10-31 15:33:08 Make corrections to the intended MIGRATOR user permissions to resolve. Run the following command in the database:
2025-10-31 15:33:08 GRANT ALL PRIVILEGES on somethingtotallynew.* to 'specify_migrator'@'%'; FLUSH PRIVILEGES;
2025-10-31 15:33:08 Creating user 'specify_user'...
2025-10-31 15:33:08 Executing: mysql -h "host.docker.internal" -P "3306" -u "root" --password="<hidden>" -e "CREATE USER 'specify_user'@'%' IDENTIFIED BY '<hidden>';"
2025-10-31 15:33:08 Granting privileges to new user...
2025-10-31 15:33:08 Executing: mysql -h "host.docker.internal" -P "3306" -u "root" --password="<hidden>" -e "GRANT SELECT, INSERT, UPDATE, DELETE, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE ON `somethingtotallynew`.* TO 'specify_user'@'%'; FLUSH PRIVILEGES;"
2025-10-31 15:33:08 Verified: 'specify_user'@'%' has required privileges on 'somethingtotallynew'.
2025-10-31 15:33:08 --------------------------------------------------
2025-10-31 15:33:08 Database and user setup complete.
2025-10-31 15:33:08 New database created: True
2025-10-31 15:33:08 New migrator user created: True
2025-10-31 15:33:08 New app user created: True
2025-10-31 15:33:08 --------------------------------------------------
2025-10-31 15:33:08 New database detected.
2025-10-31 15:33:59 Operations to perform:
2025-10-31 15:33:59   Apply all migrations: accounts, attachment_gw, auth, businessrules, contenttypes, notifications, patches, permissions, sessions, specify, workbench
2025-10-31 15:33:59 Running migrations:
2025-10-31 15:36:16   Applying specify.0001_initial... OK
2025-10-31 15:36:17   Applying accounts.0001_initial... OK
2025-10-31 15:36:17   Applying accounts.0002_auto_20211223_1206... OK
2025-10-31 15:36:19   Applying accounts.0003_auto_20220621_1541... OK
2025-10-31 15:36:20   Applying attachment_gw.0001_initial... OK
2025-10-31 15:36:20   Applying contenttypes.0001_initial... OK
2025-10-31 15:36:20   Applying contenttypes.0002_remove_content_type_name... OK
2025-10-31 15:33:20 [31/Oct/2025 15:33:20] [INFO] [specifyweb.specify.management.commands.base_specify_migration:29] Running base_specify_migration using database alias 'master'
2025-10-31 15:33:22 [31/Oct/2025 15:33:22] [INFO] [specifyweb.specify.management.commands.base_specify_migration:78] Completed using database alias 'master'
2025-10-31 15:36:21   Applying auth.0001_initial... OK
2025-10-31 15:36:21   Applying auth.0002_alter_permission_name_max_length... OK
2025-10-31 15:36:21   Applying auth.0003_alter_user_email_max_length... OK
2025-10-31 15:36:21   Applying auth.0004_alter_user_username_opts... OK
2025-10-31 15:36:21   Applying auth.0005_alter_user_last_login_null... OK
2025-10-31 15:36:21   Applying auth.0006_require_contenttypes_0002... OK
2025-10-31 15:36:21   Applying auth.0007_alter_validators_add_error_messages... OK
2025-10-31 15:36:21   Applying auth.0008_alter_user_username_max_length... OK
2025-10-31 15:36:21   Applying auth.0009_alter_user_last_name_max_length... OK
2025-10-31 15:36:21   Applying auth.0010_alter_group_name_max_length... OK
2025-10-31 15:36:21   Applying auth.0011_update_proxy_permissions... OK
2025-10-31 15:36:21   Applying auth.0012_alter_user_first_name_max_length... OK
2025-10-31 15:41:38 Traceback (most recent call last):
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/db/backends/base/base.py", line 288, in ensure_connection
2025-10-31 15:41:38     self.connect()
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
2025-10-31 15:41:38     return func(*args, **kwargs)
2025-10-31 15:41:38            ^^^^^^^^^^^^^^^^^^^^^
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/db/backends/base/base.py", line 269, in connect
2025-10-31 15:41:38     self.connection = self.get_new_connection(conn_params)
2025-10-31 15:41:38                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
2025-10-31 15:41:38     return func(*args, **kwargs)
2025-10-31 15:41:38            ^^^^^^^^^^^^^^^^^^^^^
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/db/backends/mysql/base.py", line 247, in get_new_connection
2025-10-31 15:41:38     connection = Database.connect(**conn_params)
2025-10-31 15:41:38                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/MySQLdb/__init__.py", line 123, in Connect
2025-10-31 15:41:38     return Connection(*args, **kwargs)
2025-10-31 15:41:38            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/MySQLdb/connections.py", line 185, in __init__
2025-10-31 15:41:38     super().__init__(*args, **kwargs2)
2025-10-31 15:41:38 MySQLdb.OperationalError: (1045, "Access denied for user 'specify_user'@'localhost' (using password: YES)")
2025-10-31 15:41:38 
2025-10-31 15:41:38 The above exception was the direct cause of the following exception:
2025-10-31 15:41:38 
2025-10-31 15:41:38 Traceback (most recent call last):
2025-10-31 15:41:38   File "/opt/specify7/manage.py", line 25, in <module>
2025-10-31 15:41:38     execute_from_command_line(sys.argv)
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
2025-10-31 15:41:38     utility.execute()
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/core/management/__init__.py", line 436, in execute
2025-10-31 15:41:38     self.fetch_command(subcommand).run_from_argv(self.argv)
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/core/management/base.py", line 412, in run_from_argv
2025-10-31 15:41:38     self.execute(*args, **cmd_options)
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/core/management/base.py", line 458, in execute
2025-10-31 15:41:38     output = self.handle(*args, **options)
2025-10-31 15:41:38              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/core/management/base.py", line 106, in wrapper
2025-10-31 15:41:38     res = handle_func(*args, **kwargs)
2025-10-31 15:41:38           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/core/management/commands/migrate.py", line 356, in handle
2025-10-31 15:41:38     post_migrate_state = executor.migrate(
2025-10-31 15:41:38                          ^^^^^^^^^^^^^^^^^
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/db/migrations/executor.py", line 135, in migrate
2025-10-31 15:41:38     state = self._migrate_all_forwards(
2025-10-31 15:41:38             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
2025-10-31 15:41:38     state = self.apply_migration(
2025-10-31 15:41:38             ^^^^^^^^^^^^^^^^^^^^^
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
2025-10-31 15:41:38     state = migration.apply(state, schema_editor)
2025-10-31 15:41:38             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/db/migrations/migration.py", line 127, in apply
2025-10-31 15:41:38     operation.database_forwards(
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
2025-10-31 15:41:38     self.code(from_state.apps, schema_editor)
2025-10-31 15:41:38   File "/opt/specify7/specifyweb/specify/migrations/0002_geo.py", line 75, in consolidated_python_django_migration_operations
2025-10-31 15:41:38     create_default_collection_types(apps)
2025-10-31 15:41:38   File "/opt/specify7/specifyweb/specify/api/utils.py", line 36, in create_default_collection_types
2025-10-31 15:41:38     code_set = set(Collection.objects.all().values_list('code', flat=True))
2025-10-31 15:41:38                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/db/models/query.py", line 398, in __iter__
2025-10-31 15:41:38     self._fetch_all()
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/db/models/query.py", line 1881, in _fetch_all
2025-10-31 15:41:38     self._result_cache = list(self._iterable_class(self))
2025-10-31 15:41:38                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/db/models/query.py", line 285, in __iter__
2025-10-31 15:41:38     for row in compiler.results_iter(
2025-10-31 15:41:38                ^^^^^^^^^^^^^^^^^^^^^^
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/db/models/sql/compiler.py", line 1513, in results_iter
2025-10-31 15:41:38     results = self.execute_sql(
2025-10-31 15:41:38               ^^^^^^^^^^^^^^^^^
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/db/models/sql/compiler.py", line 1560, in execute_sql
2025-10-31 15:41:38     cursor = self.connection.cursor()
2025-10-31 15:41:38              ^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
2025-10-31 15:41:38     return func(*args, **kwargs)
2025-10-31 15:41:38            ^^^^^^^^^^^^^^^^^^^^^
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/db/backends/base/base.py", line 329, in cursor
2025-10-31 15:41:38     return self._cursor()
2025-10-31 15:41:38            ^^^^^^^^^^^^^^
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/db/backends/base/base.py", line 305, in _cursor
2025-10-31 15:41:38     self.ensure_connection()
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
2025-10-31 15:41:38     return func(*args, **kwargs)
2025-10-31 15:41:38            ^^^^^^^^^^^^^^^^^^^^^
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/db/backends/base/base.py", line 287, in ensure_connection
2025-10-31 15:41:38     with self.wrap_database_errors:
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/db/utils.py", line 91, in __exit__
2025-10-31 15:41:38     raise dj_exc_value.with_traceback(traceback) from exc_value
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/db/backends/base/base.py", line 288, in ensure_connection
2025-10-31 15:41:38     self.connect()
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
2025-10-31 15:41:38     return func(*args, **kwargs)
2025-10-31 15:41:38            ^^^^^^^^^^^^^^^^^^^^^
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/db/backends/base/base.py", line 269, in connect
2025-10-31 15:41:38     self.connection = self.get_new_connection(conn_params)
2025-10-31 15:41:38                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
2025-10-31 15:41:38     return func(*args, **kwargs)
2025-10-31 15:41:38            ^^^^^^^^^^^^^^^^^^^^^
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/django/db/backends/mysql/base.py", line 247, in get_new_connection
2025-10-31 15:41:38     connection = Database.connect(**conn_params)
2025-10-31 15:41:38                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/MySQLdb/__init__.py", line 123, in Connect
2025-10-31 15:41:38     return Connection(*args, **kwargs)
2025-10-31 15:41:38            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-31 15:41:38   File "/opt/specify7/ve/lib/python3.12/site-packages/MySQLdb/connections.py", line 185, in __init__
2025-10-31 15:41:38     super().__init__(*args, **kwargs2)
2025-10-31 15:41:38 django.db.utils.OperationalError: (1045, "Access denied for user 'specify_user'@'localhost' (using password: YES)")
2025-10-31 15:41:41 Watching for file changes with StatReloader
2025-10-31 15:41:43 Exception in thread django-main-thread:
2025-10-31 15:41:43 Traceback (most recent call last):
2025-10-31 15:41:43   File "/opt/specify7/ve/lib/python3.12/site-packages/django/db/backends/base/base.py", line 288, in ensure_connection
2025-10-31 15:41:43     self.connect()
2025-10-31 15:41:43   File "/opt/specify7/ve/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
2025-10-31 15:41:43     return func(*args, **kwargs)
2025-10-31 15:41:43            ^^^^^^^^^^^^^^^^^^^^^
2025-10-31 15:41:43   File "/opt/specify7/ve/lib/python3.12/site-packages/django/db/backends/base/base.py", line 269, in connect
2025-10-31 15:41:43     self.connection = self.get_new_connection(conn_params)
2025-10-31 15:41:43                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-31 15:41:43   File "/opt/specify7/ve/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
2025-10-31 15:41:43     return func(*args, **kwargs)
2025-10-31 15:41:43            ^^^^^^^^^^^^^^^^^^^^^
2025-10-31 15:41:43   File "/opt/specify7/ve/lib/python3.12/site-packages/django/db/backends/mysql/base.py", line 247, in get_new_connection
2025-10-31 15:41:43     connection = Database.connect(**conn_params)
2025-10-31 15:41:43                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-31 15:41:43   File "/opt/specify7/ve/lib/python3.12/site-packages/MySQLdb/__init__.py", line 123, in Connect
2025-10-31 15:41:43     return Connection(*args, **kwargs)
2025-10-31 15:41:43            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-31 15:41:43   File "/opt/specify7/ve/lib/python3.12/site-packages/MySQLdb/connections.py", line 185, in __init__
2025-10-31 15:41:43     super().__init__(*args, **kwargs2)
2025-10-31 15:41:43 MySQLdb.OperationalError: (1045, "Access denied for user 'specify_user'@'localhost' (using password: YES)")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: 📋Back Log

5 participants