We have presented several examples as part of this migration blog post series. They started very simple and have been increasing in complexity. Until now, we have been rather optimistic. Get the sample code, install any module dependency, enable the module that defines the migration, and execute it assuming everything works on the first try. But Drupal migrations often involve a bit of trial and error. At the very least, it is an iterative process. Today we are going to talk about what happens after import and rollback operations, how to recover from a failed migration, and some tips for writing definition files.
Importing and rolling back migrations
When working on a migration project, it is common to write many migration definition files. Even if you were to have only one, it is very likely that your destination will require many field mappings. Running an import operation to get the data into Drupal is the first step. With so many moving parts, it is easy not to get the expected results on the first try. When that happens, you can run a rollback operation. This instructs the system to revert anything that was introduced when the migration was initially imported. After rolling back, you can make changes to the migration definition file and rebuild Drupal’s cache for the system to pick up your changes. Finally, you can do another import operation. Repeat this process until you get the results you expect. The following code snippet shows a basic Drupal migration workflow:
# 1) Run the migration.
$ drush migrate:import udm_subfields
# 2) Rollback migration because the expected results were not obtained.
$ drush migrate:rollback udm_subfields
# 3) Change the migration definition file.
# 4) Rebuild caches for changes to be picked up.
$ drush cache:rebuild
# 5) Run the migration again
$ drush migrate:import udm_subfieldsThe example above assumes you are using Drush to run the migration commands. Specifically, the commands provided by Migrate Run or Migrate Tools. You pick one or the other, but not both as the commands provided for two modules are the same. If you were to have both enabled, they will conflict with each other and fail.
Another thing to note is that the example uses Drush 9. There were major refactorings between versions 8 and 9 which included changes to the name of the commands. Finally, udm_subfields is the id of the migration to run. You can find the full code in this article.
Tip: You can use Drush command aliases to write shorter commands. Type drush [command-name] --help for a list of the available aliases.
Technical note: To pick up changes to the definition file, you need to rebuild Drupal’s caches. This is the procedure to follow when creating the YAML files using Migrate API core features and placing them under the migrations directory. It is also possible to define migrations as configuration entities using the Migrate Plus module. In those cases, the YAML files follow a different naming convention and are placed under the config/install directory. For picking up changes, in this case, you need to sync the YAML definition using configuration management workflows. This will be covered in a future entry.
Stopping and resetting migrations
Sometimes, you do not get the expected results due to an oversight in setting a value. On other occasions, fatal PHP errors can occur when running the migration. The Migrate API might not be able to recover from such errors. For example, using a non-existent PHP function with the callback plugin. Give it a try by modifying the example in this article. When these errors happen, the migration is left in a state where no import or rollback operations could be performed.
You can check the state of any migration by running the drush migrate:status command. Ideally, you want them in Idle state. When something fails during import or rollback, you would get the Importing or Rolling back states. To get the migration back to Idle, you stop the migration and reset its status. The following snippet shows how to do it:
# 1) Run the migration.
$ drush migrate:import udm_process_intro
# 2) Some non recoverable error occurs. Check the status of the migration.
$ drush migrate:status udm_process_intro
# 3) Stop the migration.
$ drush migrate:stop udm_process_intro
# 4) Reset the status to idle.
$ drush migrate:reset-status udm_process_intro
# 5) Rebuild caches for changes to be picked up.
$ drush cache:rebuild
# 6) Rollback migration because the expexted results were not obtained.
$ drush migrate:rollback udm_process_intro
# 7) Change the migration definition file.
# 8) Rebuild caches for changes to be picked up.
$ drush cache:rebuild
# 9) Run the migration again.
$ drush migrate:import udm_process_introTip: The errors thrown by the Migrate API might not provide enough information to determine what went wrong. An excellent way to familiarize yourselves with the possible errors is by intentionally breaking working migrations. In the example repository of this series, there are many migrations you can modify. Try anything that comes to mind: not leaving a space after a colon (:) in a key-value assignment; not using proper indentation; using wrong subfield names; using invalid values in property assignments; etc. You might be surprised by how Migrate API deals with such errors. Also, note that many other Drupal APIs are involved. For example, you might get a YAML file parse error, or an Entity API save error. When you have seen an error before, it is usually faster to identify the cause and fix it in the future.
What happens when you rollback a Drupal migration?
In an ideal scenario, when a migration is rolled back, it cleans after itself. That means, it removes any entity that was created during the import operation: nodes, taxonomy terms, files, etc. Unfortunately, that is not always the case. It is very important to understand this when planning and executing migrations. For example, you might not want to leave taxonomy terms or files that are no longer in use. Whether any dependent entity is removed or not has to do with how plugins or entities work.
For example, when using the file_import or image_import plugins provided by Migrate File, the created files and images are not removed from the system upon rollback. When using the entity_generate plugin from Migrate Plus, the created entity also remains in the system after a rollback operation.
In the next blog post, we are going to start talking about migration dependencies. What happens with dependent migrations (e.g., files and paragraphs) when the migration for host entity (e.g., node) is rolled back? In this case, the Migrate API will perform an entity delete operation on the node. When this happens, referenced files are kept in the system, but paragraphs are automatically deleted. For the curious, this behavior for paragraphs is actually determined by its module dependency: Entity Reference Revisions. We will talk more about paragraphs migrations in future blog posts.
The moral of the story is that the behavior migration system might be affected by other Drupal APIs. And in the case of rollback operations, make sure to read the documentation or test manually to find out when migrations clean after themselves and when they do not.
Note: The focus of this section was content entity migrations. The general idea can be applied to configuration entities or any custom target of the ETL process.
Re-import or update migrations
We just mentioned that Migrate API issues an entity delete action when rolling back a migration. This has another important side effect. Entity IDs (nid, uid, tid, fid, etc.) are going to change every time you rollback an import again. Depending on auto generated IDs is generally not a good idea. But keep it in mind in case your workflow might be affected. For example, if you are running migrations in a content staging environment, references to the migrated entities can break if their IDs change. Also, if you were to manually update the migrated entities to clean up edge cases, those changes would be lost if you rollback and import again. Finally, keep in mind test data might remain in the system, as described in the previous section, which could find its way to production environments.
An alternative to rolling back a migration is to not execute this operation at all. Instead, you run an import operation again using the update flag. This tells the system that in addition to migrating unprocessed items from the source, you also want to update items that were previously imported using their current values. To do this, the Migrate API relies on source identifiers and map tables. You might want to consider this option when your source changes overtime, when you have a large number of records to import, or when you want to execute the same migration many times on a schedule.
Note: On import operations, the Migrate API issues an entity save action.
Tips for writing Drupal migrations
When working on migration projects, you might end up with many migration definition files. They can set dependencies on each other. Each file might contain a significant number of field mappings. There are many things you can do to make Drupal migrations more straightforward. For example, practicing with different migration scenarios and studying working examples. As a reference to help you in the process of migrating into Drupal, consider these tips:
- Start from an existing migration. Look for an example online that does something close to what you need and modify it to your requirements.
- Pay close attention to the syntax of the YAML file. An extraneous space or wrong indentation level can break the whole migration.
- Read the documentation to know which source, process, and destination plugins are available. One might exist already that does exactly what you need.
- Make sure to read the documentation for the specific plugins you are using. Many times a plugin offer optional configurations. Understand the tools at your disposal and find creative ways to combine them.
- Look for contributed modules that might offer more plugins or upgrade paths from previous versions of Drupal. The Migrate ecosystem is vibrant, and lots of people are contributing to it.
- When writing the migration pipeline, map one field at a time. Problems are easier to isolate if there is only one thing that could break at a time.
- When mapping a field, work on one subfield at a time if possible. Some field types like images and addresses offer many subfields. Again, try to isolate errors by introducing individual changes each time.
- There is no need to do every data transformation using the Migrate API. When there are edge cases, you can manually update those after the automated migration is completed. That is, no more rollback operations. You can also clean up the source data in advance to make it easier to process in Drupal.
- Commit to your code repository any and every change that produces the right results. That way, you can go back in time and recover a partially working migration.
- Learn about debugging migrations. We will talk about this topic in a future blog post.
- Seek help from the community. Migrate maintainers and enthusiasts are very active and responsive in the #migrate channel of Drupal slack.
- If you feel stuck, take a break from the computer and come back to it later. Resting can do wonders in finding solutions to hard problems.
What did you learn in today’s blog post? Did you know what happens upon importing and rolling back a migration? Did you know that in some cases, data might remain in the system even after rollback operations? Do you have a use case for running migrations with the update flag? Do you have any other advice on writing migrations? Please share your answers in the comments. Also, I would be grateful if you shared this blog post with your colleagues.
Next: Migrating files and images into Drupal
This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors. Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.
Turning off links to entities that have been Rabbit Holed in modern Drupal
You have used Rabbit Hole module to prevent people from being able to directly visit to select content items, but when you include those items in a reference list Drupal is still linking to them.
Here is one way to fix that.
Scaling Community Conversations and Decisions presentation @ FluConf
How can a group of thousands of people talk about and decide anything? How's this 'community' concept supposed to work at scale, even in theory?
Any Free/Libre Open Source Software project will have elements of do-ocracy (rule of those who do the work), but not all decisions should devolve to implementors. A better ideal is that decisions should be made by the people who are most affected.
Is Drutopia right for you? Drutopia versus other low-cost options
Building your website on Drutopia has the advantage of allowing multiple people to create content of different defined types that can be clearly related related to one another, listed and presented in different ways, and filtered.
When There's Not a Module for That: How to Make (and Maintain) a New Module
You've built sites with Drupal and know that with a few dozen modules (and a ton of configuring), you can do nearly everything in modern Drupal.
But what do you do when there's not a module for that? When the modules that exist don't meet your needs, and cannot be made to by contributing changes?
You make your own.
This blog post accompanies helps you take that step. Making a module is something that anyone can do. Every person developing a module is still learning.
Community-managed categories is an idea from just about the beginning of my time as a web developer. As "Community-managed taxonomy" it was my submission to the 2007 Summer of Code, barely a couple years into my time as a Drupal developer.
The Drupal module Community Managed Taxonomy, or CMT, variously known as Community Managed Categories, seeks to bring the possibility of mass participation to categorization (taxonomy) and therefore potentially site structure.
As the project page put it:
Community-managed taxonomy (CMT) opens categorization of content to the site's community. Users can influence both what terms nodes are tagged with and how these terms are themselves organized.
It can also be used to make structured tags on the fly. Users do not need to be logged in to make or propose terms for content.
I very much hope to get back to this work. I never did get the module working—Agaric was a new company and my father became ill and was killed by the hospital that summer, and my mentor and i were not the best match—and after failing to complete the module and get the second stipend, the time and circumstance has not yet returned. Please contact us if you may be able to help make the time right in 2021!
This capability to allow a large community to coordinate in categorizing may be more salient when democratically-moderated mass communication is made possible. That's a goal i have pursued even longer, and i think it is more important and possibly logically prior to community-managed taxonomy. First we need community-managed communication, so that the resources we build together can reach, be distributed to, the people who should know. But building community-managed communication may bring up the need for community-managed categories directly, too— how do we decide what the groups are that we can manage communication within? That's a job for community-managed taxonomy, probably.
It is all tied up to what I have been aiming for for decades. This description is from 2008:
People have always needed something better than mailing lists— or other communication tools as they exist now. We need something that can reach millions of people (or billions– everyone) and still be open to everyone on an equal basis. Reaching everyone means filtering to reduce quantity and increase quality. Staying open to everyone means that the filtering must not be controlled by any group, must in some true sense belong to everyone.
The Internet has this potential. (The issue of access remains crucial, but is separate from helping the Internet reach closer to its potential for people who do have access.)
People Who Give a Damn is incorporated as a nonprofit organization to connect, without interference and without wasting anyone's time, everyone who gives a damn.
As the first technique to achieve this, anyone signed up to receive messages in a network can submit a message to be sent. The message will be publicly available immediately, but it will be moderated by a random sample of other people in the network pulled to serve jury duty. If they decide it is important enough to send to everyone in the network, it is sent to everyone. If not, the message will have more limited distribution to the sender's personal contacts and possibly to groups within the overall network to which the sender belongs.
This is simple. Yet it will make possible horizontal communication, not top-down few-to-many broadcasts, that is also mass communication. We need horizontal mass communication because we need mass cooperation and collaboration. It is possible with current technology, and necessary for the well-being of ourselves, our friends and family, our fellow humans, our Earth. Anyone interested in updates on progress or how they can help, please contact me.
A program is free software if the program's users have the four essential freedoms:
- The freedom to run the program as you wish, for any purpose (freedom 0).
- The freedom to study how the program works, and change it, so it does your computing as you wish (freedom 1).
- The freedom to redistribute copies so you can help others (freedom 2).
- The freedom to distribute copies of your modified versions to others (freedom 3).
Access to the source code is a precondition for this. To determine whether the source code is free, see what license it carries and then check the GNU project's list of licenses.
We do a few tasks that we do not have free software for. So, we use non-free programs for them. Sometimes we use specific non-free software that a client insists on including in the web site. By using non-free programs, we sacrifice some of our freedom; some activists would refuse to do that. We do not like the compromise, so we help develop free replacements for those programs. Whenever possible, we use software that reflects our values.
Operating Systems
GNU/Linux
We have chosen to use GNU/Linux as our default system for our local development. When we take on a new student, we install a GNU/Linux distribution. We always give the option of installing a different distribution, or if a student wishes to do so, they may. These are the favored GNU/Linux distributions in use by our cooperative team *members:
These are not the best versions of GNU/Linux in regards to being completely free; you should consult the list of free distributions on the Free Software Foundation website.
* Currently one team member is using the proprietary but BSD-based Mac OS X, which is compliant with the Unix 03 / POSIX standard which GNU/Linux distributions also meet.
Browsers
Firefox
As developers, we have to test client sites in all browsers, but for working and building sites, we use Mozilla Firefox. Although the source code of Firefox is free software, they include some icons, trademarks, and logos in the download that make it non-free. You can easily remove these as has been done with IceCat, the GNU version of the Firefox browser. It has great performance, developer tools, and community. The plus side of having a community around the software we use is having access to a large pool of people with experience and guidance as we learn and build together.
Tor Browser
As citizens, we are not fond of being tracked, so we use a free anonymizing web browser that does not allow tracking. It is called Tor.
Micky uses and loves the Brave browser on Android and her GNU/Linux laptop. It blocks ADs right out of the box! https://brave.com/
Without looking deeply at Qwant, it looks pretty decent - https://www.qwant.com/
We came across Qwant via https://iridiumbrowser.de/ - a more secure chromium release, but probably has some things we may not know about or want...
File Storage and Calendar
NextCloud
NextCloud is a collection of utilities suited for the De-Googlization process. It provides most of the popular tools in Google Drive. Agaric uses a hosted version of NextCloud on MayFirst.org servers that is inclusive of:
- document and file storage
- shared document editing
- multi-device synchronization
- image galleries
- calendar
- contacts
See a comparison of the features NextCloud offers vs. the proprietary options like GoogleDocs/Drive.
Finance, Accounting, and Bookkeeping
GNUcash
Accounting software that we use for our bookkeeping.
You can see a review of GNUcash vs. Quickbooks and decide if it works for you. We have found a few bookkeeping cooperatives that do accounting with GnuCash.
Real-time Chat
As a team, we rely on different tools to communicate with each other and with clients about daily activities and long term project goals. We have a distributed team at locations around the world and must maintain contact especially when pair-programming or during a migration which calls for all-hands-on-deck, as well as sharing some long informational text notes and documents that include administrative information.
Freenode
IRC - Internet Relay Chat - Realtime Text Chat: Yes, we still use IRC, and you can find us on irc.freenode.net server in the #devs-r-us channel
Our preferences here are as varied as our team members: some use irssi via a remote, always-on virtual server, many use desktop clients, such as HexChat or Konversation, and still, others prefer the web-based solution "The Lounge."
MayFirst.org hosts Agaric.com email
Email Client: Thunderbird
An email client from Mozilla, which also makes Firefox, and is available for your phone. It also has an encryption plugin called EnigMail that works well and is not super tricky to get set up.
Hosted Email: RiseUp: Encrypted services run by anonymous volunteers and you must be invited to have a membership.
MayFirst offers three web-based email solutions.
- Roundcube which has a friendly and simple web interface, making it the easier of the two programs to use.
- SquirrelMail is an option that is Javascript-free!
- Horde, on the other hand, offers more than just email - you can share calendars, to-dos and more with other members of your group.
Hosted Email
Protonmail: An email service that is hosted and encrypted.
Email Lists:
We use email list servers for mailing lists based on groups and topics. It allows group mailing to people that sign up for a specific list.
MayFirst Email Server
RiseUp Email Server
Social Media
Mastodon: Publish anything you want: links, pictures, text, video. All on a platform that is community-owned and ad-free.
Social.coop: A community similar to Twitter, the main difference is that it is owned by the members. For as little as $1 a month you can become an owner/member and take part in shaping the future of the platform. You can find and follow Agaric in social.coop, a coop-run corner of the fediverse, a cooperative and transparent approach to operating a social platform
Live Streaming
MayFirst Live Streaming: MayFirst membership includes live streaming.
Conference Calls and Online Meetings
Some Agaric team members are using Jitsi recognizing that it is a work in progress and there may be technical failures at times - such as we have also found using Google Hangouts - lag time, cut-offs, poor sound quality and issues with screen sharing. At times we have found that we need to use a proprietary solution that seems to work reliably as we continue to support development efforts and bug fixes with Jitsi. At the heart of Jitsi are Jitsi Videobridge and Jitsi Meet, which let you have conferences on the internet, while other projects in the community enable other features such as audio, dial-in, recording, and simulcasting.
You can self-host an instance of Jitsi or choose a hosted version. You can use http://meet.jit.si or an instance is also available for public use at https://meet.mayfirst.org We do encourage you to become a MayFirst member and have access to all of the free software tools they offer. The Jitsi project needs volunteers to use and test Jitsi so it can get better swiftly!
Currently, Agaric was using and paying for, the proprietary Zoom audio/video conference call service and software. Since October we have been using BigBlueButton a free software alternative. BBB has been working well for us and we have worked it into being our go-to solution for protecting our privacy.
BigBlueButton
BigBlueButton.org: Agaric uses BigBlueButton as our video-chat meeting platform. We also offer free chatroom hosting for those that cannot afford to pay for this service.
Phone Calls and Text Messages
Signal: Agaric uses Signal to encrypt SMS text messages and phone calls. Encrypted phone and instant messaging found to be secure and recommended by Edward Snowden as the only truly encrypted messaging app that is not able to be decrypted by anyone. Note that security is an arms race and this could become false at any time.
Collaborative Note Taking
Etherpad: When hosting an online meeting we usually open a shared note pad so that everyone may contribute to getting the important bits logged. Etherpad text is synchronized as you type so that everyone viewing the page sees the same text. This allows you to collaborate on documents with large or small teams seamlessly! We use the hosted version, but you are welcome to host it yourself. We have tried a few online pads and settled on Etherpad as the most reliable.
Collaborative Ongoing Discussion
With some collaborators, particularly people involved with the Drutopia initiative, we use Mattermost rather than IRC. Mattermost can be more useful for ongoing discussions; it is similar to Slack and offers a threaded conversation. The community version is free software.
Notes and ToDo Lists
TomBoy A tiny app that lets you take notes while it conveniently makes hyperlinks out of titles and allows synchronization over SSH and more.
Password Management
KeePass A password management system that takes most of the worry, distraction and thinking out of storing and retrieving your login information for multiple projects and sites.
Text Document Editing, Spreadsheets, and Presentations
Libre Office: A suite of office tools similar to Microsoft Office, Documents, Spreadsheets, Slides. We use LibreOffice tools that come as core software in the distributions of GNU/Linux we are using. You may have heard of OpenOffice; it is now called LibreOffice. It consists of basic publishing and calculating software for doing office tasks. These are the ones we use most often:
1. LibreOffice Calc - Similar features and functions of a calculating software to make spreadsheets, such as MicroSoft Excel
2. LibreOffice Writer - Similar features and functions of a word processor such as MicroSoft Word
3. LibreOffice Impress - We use this tool to build slide decks and presentations using text/graphics and videos; it is similar to Microsoft PowerPoint in features.
Project Management and Issue Tracking
*GitLab: This tool is a web-based and self-hosted Git-repository manager with a wiki and issue-tracking features. We also use Gitlab for cooperative development on our projects.
*Although GitLab isn't fully free software, it does offer a self-hosted version that is. The Enterprise hosted version has extra features and is proprietary.
Redmine: A free program that you can run locally or on your own server for use as a project management and issue tracking tool. Before finding GitLab, we used a self-hosted instance of Redmine which is free software.
Decision Making and Voting
Loomio: A hosted service available at http://loomio.org
Loomio offers distributed decision-making system where you can make groups that can have discussions and make decisions without an in-person meeting. Decide yes or no, or that you need more information.
Note that Loomio also has built a great cooperative resource on at their other URL - http://loomio.coop
Customer Relationship Management
CiviCRM: Agaric is working with the developers at MyDropWizard to take a look at CiviCRM with Drupal 8.
CiviCRM is a free software to manage client relationships and memberships. We have not deployed it yet.
Resources and Free Software Directories
- Free Software Directory
- Awesome-list of self-hosted applications
- You can search for Free software alternatives. - be sure to search for Free/Open Source, not the free trial software, and LOOK at the license.
You can contribute to groups working towards solutions, there are many roles, and you do not have to be a developer. As an example, *IndieWeb and Jitsi are projects that we make time to support with development, testing, outreach, and feedback.
*With IndieWeb, you can take control of your articles and status messages can go to all services, not just one, allowing you to engage with everyone. Even replies and likes on other services can come back to your site, so they’re all in one place.
Framasoft: A large collection of free software tools where we use the calendar and polling software most often. We are experimenting with several other FramaSoft tools and may adopt them in the future.
If this has been a helpful read, please pass it on and let us know in the comments how it helped you. A follow-up post will list the tools we use for development purposes. Please be sure to mention any free software you have found and are using now.
I will leave you with an excellent TEDx talk where Richard Stallman explains Free Software:
Sign up to be notified when Agaric gives a migration training:
What are Agaric's initiatives, you ask?
We do much more than build websites. We build tools that work for the social change we want to see. We teach and empower others to use digital technology safely, responsibly, and effectively. We participate in building and supporting communities that are founded on democratic-ownership and knowledge-sharing. We engage in a plethora of movements, and we look for ways to bridge the gaps between them. Akin to the mushrooms after which we are named, we work across vast nutritive networks so that we can bring nourishment to our human ecosystems. This page is our feeble attempt to showcase the meaning behind Agaric Tech Cooperative.
Overview
I. Platforms and Hosted Services
II. Open Community Events
III. Contributions to Drupal
IV. Movements and Networks
V. Auxiliary Tech Projects
I. Platforms and Hosted Services
Health communities
We have been building innovative web-based homes for scientific and health communities since 2008— most recently partnering with the National Institute for Children's Health Quality.
Find It
The Find It program locator and event discovery platform for cities, towns, counties, and most any geographical community. Use granular search to locate resources that otherwise may be hard to find!
Drutopia
The Drutopia Cooperative Platform by Agaric combines the ease of use you find from software as a service website builders (like Wix and Squarespace) with the freedom and control of free (libre) software. It is LibreSaaS like Ghost(Pro) or WordPress.com, but it is built for and with grassroots groups. Importantly, the platform is collectively controlled by the people who rely on it.
Online Learning
Host your school online. Stay safely at home while learning and interacting with others. We will host your online courses and video chat without connections to Google or Facebook and free from spyware or malware that can go undetected in proprietary software. We are using free and open source solutions that are currently used by colleges and schools around the world. Agaric can build upon these solutions to customize your experience and suit your needs.
CommunityBridge
CommunityBridge is an online video conferencing service that we provide freely to trusted friends and activists in our network. It is also where we host our community events. The video conferencing software behind it is BigBlueButton, a free and open source software built for educational communities to learn together safely in a digital space without fear of being passively tracked and surveilled.
II. Open Community Events
Show & Tell
Agaric hosts a weekly online gathering for people to meet and share what they have learned. Sometimes we talk about the logistics of worker owned Cooperatives and sometimes we have technical talks and we look at code. Every week we also get to know each other better and this leads to sharing work on projects. Learn more about Show and Tell. One of our most popular meetings was a presentation and discussion on a real life condition that affects quite a few developers. This condition is called imposter syndrome.
Movie Night
Agaric hosts Movie Nights, where we support organizations in sponsoring facilitated movie-watching events and engage in deep discussions that sometimes reveal actionable steps that community members can take together to overcome social and political issues. Currently we are only able to watch movies that are hosted on Vimeo, Youtube, or DailyMotion. If you would like to suggest a movie, we will facilitate the event. Make some popcorn, grab your favorite beverage, and invite your friends!
III. Contributions to Drupal
Drupal is a free software content management system powered by one of the largest communities in the software world— very much including Agaric.
Contributed Drupal Modules
We contribute to Drupal core and more than 100 modules that extend its functionality. All of these useful projects are free for anyone to download, use or modify for their own needs.
Trainings
We teach development teams and solo coders in-person or online, with the program tailored to the problems you are trying to solve. We have practical experience in developing a multitude of web sites, migrating content, and running technology projects - we love to learn and to teach. We will impart the knowledge and skills you need to get work done, and done right.
31 Days of Migrations
Mauricio's epic month of migration tutorials is an expansive resource to teach you how to perform Drupal migrations. At some point, almost every website will need to be migrated to a newer or more secure platform, or to a platform with new and different features. Migrating to a new and different platform or server happens for many reasons during the life of a project. Be prepared!
Drutopia
Drutopia is a flexible content management system with many features built specially for grassroots organizations. It already helps groups share goals, call for action, collect donations, and report progress. Most important, Drutopia's developers seek to design ongoing improvements and capabilities with groups organizing for a better world.
"Wirth Co-op won't reopen", a tiny heading on the front page of North News announced. The story ran inside on page four.
The grocery store, right in my neighborhood, set out to build community wealth and provide healthy food in north Minneapolis. The loss of its promise is crushing.
Wirth Grocery's failure to act as a cooperative—with its own existence at stake—is even more upsetting. Cooperatives are one of the few spaces where people have a chance to take collective control of something that matters to them.
One epic month of blog post tutorials about migrating into Drupal 8 and 9 by Mauricio who also is the driving force behind Agaric's migration trainings and Agaric's upgrade services.
One man. One month. Many migrations.
The ever intensifying climate crisis is an existential threat. It was brought upon by people designing and building technology reliant upon the extraction and burning of fossil fuel. That and an economic system reliant on ever increasing consumption, demanding energy production unsustainable to the planet.
From September 20th through September 27th millions of people across the globe skipped school, walked out of workplaces, and joined in the streets to demand bold, swift action for climate justice. Many in the tech industry, including ourselves, went on digital strike, shuttering our websites for the day to join in the action. In fact, we partnered with 350.org to improve the mapping tool used to help people organize and find climate strike actions.
But what does bold, swift action mean? What is required of us to respond to the enormity of this climate catastrophe?
One thing is for certain - it won't be fixed with the same values, systems and forces that ushered in this emergency.
There are many ideas out there on what we should do, with many different names. Whatever the specifics, the path to climate justice rests on our ability to move away from extractive relationships with our earth and quickly grow regenerative relationships instead. Many are calling this the Commons Transition. Those of us working in the tech industry have tremendous influence in participating in and assisting with this transition.
What is the Commons?
The Commons is a way to organize and manage resources collaboratively among the community of producers and users. They exist outside of both the public and private sector. The commons is flourishing all around us, particularly where indigenous communities have been able to defend themselves and their land. The Zapatista caracoles in Chiapas, Mexico, the confederalist communes of Rojava, and the Potato Park in Peru are just a few examples. All are a blend of retaining generations-long wisdom of living in right relation, an unlearning of the extractive patterns that have gained dominance, and innovating new ways of commoning.
Free and open-source software movements put into practice much of the Commons' values. Combining this approach to software development with cooperative, community-based economic models can help us build technology that is sustainable and helps facilitate and defend the commons.
The Tech Industry's Role in the Commons Transition
As mentioned, the commons is all around us. Yet these commons are constantly battling enclosure, and many more commons projects need our support to take hold. The first step is to take stock of the commons we are currently part of. Millions of us are already members of credit unions, food cooperatives, worker-owners in cooperatives, purchasers of cooperatively produced goods and visitors to cooperatively managed parks and open spaces.
This morning, I walked to the Westwood Food Cooperative to buy produce and am now typing this essay at Kahlo's, a Mexican-American family-run restaurant with a plant-heavy menu. I am a worker-owner of Agaric, a tech cooperative. We're also members of MayFirst, a cooperative itself which provides the hosting infrastructure for our clients. The commons are everywhere.
How might we better support the commons we are already part of?
The best way to find is to ask. To find commons near you, visit SolidarityEconomy.us. It's not comprehensive, but it's a good start.
Once better connected to the commons you are part of, examine what other aspects of your life can align with the commons. If you tend to shop at a big box store, look into local food co-ops and community supported agriculture (CSA). If you work for an agency, consider how you might transition it to a cooperative with a social-mission prioritizing sustainable practices. If you don't see your company moving in a democratic direction, you can still shift it that way by unionizing. These are tall orders, but then again, these times call for tall orders.
Lead By Example, But Towards Systemic Change
The quickest way to effect change is to work within our spheres of influence. But one person shopping at a food co-op instead of Wal-Mart is just a drop in the bucket. How do we make our individual actions add up to the systemic change we need? In the essay "We Can't Do It Ourselves,"
published in Low Tech Magazine, Kris De Decker explains,
A sustainability policy that focuses on systemic issues reframes the question from “how do we change individuals’ behaviours so that they are more sustainable?” to “how do we change the way society works?”.
De Decker elaborates that,
Addressing the sociotechnical underpinnings of “behaviour” involves attempting to create new infrastructures and institutions that facilitate sustainable lifestyles, attempting to shift cultural conventions that underpin different activities, and attempting to encourage new competences that are required to perform new ways of doing things.
Low Tech Magazine is leading by example by running their website completely off of solar power they themselves produce. I encourage fellow designers and developers to check out the site and the detailed write up on the meticulous, thorough work they did to minimize the site's energy usage and rig up solar panels to power the server the site runs on.
Similarly, however we can, as tech workers we should be building our technology sustainably and in service of regenerative relations.
Use and Demand Renewable Energy
We must move away from fossil fuels and towards clean, renewable energy. For Agaric, this means getting the hosting providers we partner with to use clean, renewable energy. Most of our sites run on MayFirst, a tech cooperative we are members and leaders within. I started a discussion thread to transition our tech stack to clean, renewable sources . The discussion has become an initiative. Our work is just getting started, but this shows the benefit of meeting our needs democratically.
You can see where your hosting providers stand by using https://ecograder.com
If your host isn't on 100% renewables, talk to them and work to get them there.
You can also choose to go with a hosting provider that's already green, such as https://greenhost.net
Design for Energy Efficiency
Web development trends have lead to massive growth in the size and resource usage of websites. There are counter trends appearing in response - static site generators, lean content strategies and sustainable design. Just as we care about the energy efficiency of our cars, homes and appliances we should care too about the carbon footprint of our websites and apps. Plus, simpler, efficient online tools facilitate easier maintenance and faster load times for users (especially important for those on limited data plans and older devices). As contributors and proponents of Drupal, there is certainly room for improvement within our practice for more sustainable design.
Divest, Invest for the Transition
Technology is not neutral. What it is put into service of has tremendous impact. The climate strike demands of the Tech Workers Coalition are a great starting point.
- Zero carbon emissions by 2030
- Zero contracts with fossil fuel companies
- Zero funding of climate denial lobbying or other efforts
- Zero harm to climate refugees and frontline communities
Work to formalize these principles at your own workplace and make them public. If you have contracts with fossil fuel companies or climate deniers work to end them. A great group to link up with around this is ClimateAction.tech
Unionize to Build Counter Power
Not all of us work in democratic workplaces. For most of us, creating change in hierarchical startups and corporations is more difficult. Most companies are oriented towards shareholder needs, not worker or planet needs. Unions build counter power, forcing business to take our needs into account.
This is a daunting task. However, worker self-organizing is on the rise, especially in our industry. What starts as an internal petition signed and sent to management, can build to lasting formations that can shift companies to align with the values we need to make the transition.
Joining the Campaign to Organize Digital Employees is a great way to get started.
Conclusion
Transitions aren't easy. They're uncomfortable, they require risk and there's no guarantee of success. Still, fixing our sights on a world of care, balance and right relation with the earth and then making it so, in our home and in our workplace is deeply rewarding.
Industry audit and competitive review
You may in many ways be without peer, but there are always competitors for the attention of your audience. Identifying top peers and reviewing their respective content helps you get a wider perspective both on what potential listeners, members, and donors will be seeing and what seems to be working for others— we can start thinking together about where to emulate and where to differentiate, informing all of our work together.
Content strategy
Building on the review of peers, Agaric will work with you to briefly interview current and potential clients and develop personas and user stories.
Content style guide
Along with bringing consistency to cooperative output (and saving time sweating the details every time they come up), a good content (copywriting) style guide incorporates suggestions for clear and effective writing and helps your unique aspects shine through. It can help tell your story in a consistent way and help let your individual personalities show through while maintaining collective coherence.
User research and testing
Agaric applies a Lean UX Research methodology to answer critical user experience questions with relevant, meaningful, and actionable data.
From reviewing your goals and audiences we recommend answering the following questions:
- What prevents users from becoming paying subscribers or sustaining donor members on the site?
- What features would enable users to invite friends to become supporters?
- Who is most engaged? least engaged?
- Which conversion goals are being met? unmet?
- What new audience do we want to reach?
We recommend and use the following research and testing approaches:
- A/B testing
- Brand audit
- Browser and device analysis
- Competitive analysis
- Content audit
- Contextual inquiry
- Heat map analysis
- Heuristic evaluation
- Site analytics review
- SWOT (Strengths, Weaknesses, Opportunities, Threats) analysis
- Usability testing
- User interviews
- User surveys
Not all will fit the purpose or budget of every part of a project, but good insights into what to build and why is more valuable than simply building well and quickly.
User testing & revisions
We always recommend at least one round dedicated to measuring and improving. Using analytics and user tests, we identify what is working, what is not and needs to be changed, and what is missing and needs to be built. We then build on the previous work to do the fixes and enhancements with the highest expected impact.
Today we will learn how to migrate content from Microsoft Excel and LibreOffice Calc files into Drupal using the Migrate Spreadsheet module. We will give instructions on getting the module and its dependencies. Then, we will present how to configure the module for spreadsheets with or without a header row. There are two example migrations: images and paragraphs. Let’s get started.
Getting the code
You can get the full code example at https://github.com/dinarcon/ud_migrations.
The module to enable, as in yesterday's post in which we imported Google Sheets, is UD Google Sheets, Microsoft Excel, and LibreOffice Calc source migration whose machine name is ud_migrations_sheets_sources. It comes with four migrations: udm_google_sheets_source_node.yml, udm_libreoffice_calc_source_paragraph.yml, udm_microsoft_excel_source_image.yml, and udm_backup_csv_source_node.yml. The image migration uses a Microsoft Excel file as source. The paragraph migration uses a LibreOffice Calc file as source. The CSV migration is a backup in case the Google Sheet is not available. To execute the last one you would need the Migrate Source CSV module.
You can get the Migrate Spreadsheets module using composer: composer require drupal/migrate_spreadsheet:^1.0. This module depends on the PHPOffice/PhpSpreadsheet library and many PHP extensions including ext-zip. Check this page for a full list of dependencies. If any required extension is missing the installation will fail. If your Drupal site is not composer-based, you will not be able to use Migrate Spreadsheet, unless you jump through a lot of hoops.
Understanding the example set up
This migration will reuse the same configuration from the introduction to paragraph migrations example. Refer to that article for details on the configuration. The destinations will be the same content type, paragraph type, and fields. The source will be changed in today's example, as we use it to explain Microsoft Excel and LibreOffice Calc migrations. The end result will again be nodes containing an image and a paragraph with information about someone’s favorite book. The major difference is that we are going to read from different sources.
Note: You can literally swap migration sources without changing any other part of the migration. This is a powerful feature of ETL frameworks like Drupal’s Migrate API. Although possible, the example includes slight changes to demonstrate various plugin configuration options. Also, some machine names had to be changed to avoid conflicts with other examples in the demo repository.
Understanding the source document and plugin configuration
In any migration project, understanding the source is very important. For Microsoft Excel and LibreOffice Calc migrations, the primary thing to consider is whether or not the file contains a row of headers. Also, a workbook (file) might contain several worksheets (tabs). You can only migrate from one worksheet at a time. The example documents have two worksheets: UD Example Sheet and Do not peek in here. We are going to be working with the first one.
The spreadsheet source plugin exposes seven configuration options. The values to use might change depending on the presence of a header row, but all of them apply for both types of document. Here is a summary of the available configurations:
fileis required. It stores the path to the document to process. You can use a relative path from the Drupal root, an absolute path, or stream wrappers.worksheetis required. It contains the name of the one worksheet to process.header_rowis optional. This number indicates which row containing the headers. Contrary to CSV migrations, the row number is not zero-based. So, set this value to1if headers are on the first row,2if they are on the second, and so on.originis optional and defaults toA2. It indicates which non-header cell contains the first value you want to import. It assumes a grid layout and you only need to indicate the position of the top-left cell value.columnsis optional. It is the list of columns you want to make available for the migration. In case of files with a header row, use those header values in this list. Otherwise, use the default title for columns:A,B,C, etc. If this setting is missing, the plugin will return all columns. This is not ideal, especially for very large files containing more columns than needed for the migration.row_index_columnis optional. This is a special column that contains the row number for each record. This can be used as unique identifier for the records in case your dataset does not provide a suitable value. Exposing this special column in the migration is up to you. If so, you can come up with any name as long as it does not conflict with header row names set in thecolumnsconfiguration. Important: this is an autogenerated column, not any of the columns that come with your dataset.keysis optional and, if not set, it defaults to the value ofrow_index_column. It contains an array of column names that uniquely identify each record. For files with a header row, you can use the values set in thecolumnsconfiguration. Otherwise, use default column titles likeA,B,C, etc. In both cases, you can use therow_index_columncolumn if it was set. Each value in the array will contain database storage details for the column.
Note that nowhere in the plugin configuration you specify the file type. The same setup applies for both Microsoft Excel and LibreOffice Calc files. The library will take care of detecting and validating the proper type.
Migrating spreadsheet files with a header row
This example is for the paragraph migration and uses a LibreOffice Calc file. The following snippets shows the UD Example Sheet worksheet and the configuration of the source plugin:
book_id, book_title, Book author
B10, The definitive guide to Drupal 7, Benjamin Melançon et al.
B20, Understanding Drupal Views, Carlos Dinarte
B30, Understanding Drupal Migrations, Mauricio Dinarte
source:
plugin: spreadsheet
file: modules/custom/ud_migrations/ud_migrations_sheets_sources/sources/udm_book_paragraph.ods
worksheet: 'UD Example Sheet'
header_row: 1
origin: A2
columns:
- book_id
- book_title
- 'Book author'
row_index_column: 'Document Row Index'
keys:
book_id:
type: stringThe name of the plugin is spreadsheet. Then you use the file configuration to indicate the path to the file. In this case, it is relative to the Drupal root. The UD Example Sheet is set as the worksheet to process. Because the first row of the file contains the header rows, then header_row is set to 1 and origin to A2.
Then specify which columns to make available to the migration. In this case, we listed all of them so this setting could have been left unassigned. It is better to get into the habit of being explicit about what you import. If the file were to change and more columns were added, you would not have to update the file to prevent unneeded data to be fetched. The row_index_column is not actually used in the migration, but it is set to show all the configuration options in the example. The values will be 1, 2, 3, etc. Finally, the keys is set the column that serves as unique identifiers for the records.
The rest of the migration is almost identical to the CSV example. Small changes were made to prevent machine name conflicts with other examples in the demo repository. For reference, the following snippet shows the process and destination sections for the LibreOffice Calc paragraph migration.
process:
field_ud_book_paragraph_title: book_title
field_ud_book_paragraph_author: 'Book author'
destination:
plugin: 'entity_reference_revisions:paragraph'
default_bundle: ud_book_paragraphMigrating spreadsheet files without a header row
Now let’s consider an example of a spreadsheet file that does not have a header row. This example is for the image migration and uses a Microsoft Excel file. The following snippets shows the UD Example Sheet worksheet and the configuration of the source plugin:
P01, https://agaric.coop/sites/default/files/pictures/picture-15-1421176712.jpg
P02, https://agaric.coop/sites/default/files/pictures/picture-3-1421176784.jpg
P03, https://agaric.coop/sites/default/files/pictures/picture-2-1421176752.jpg
source:
plugin: spreadsheet
file: modules/custom/ud_migrations/ud_migrations_sheets_sources/sources/udm_book_paragraph.ods
worksheet: 'UD Example Sheet'
header_row: 1
origin: A2
columns:
- book_id
- book_title
- 'Book author'
row_index_column: 'Document Row Index'
keys:
book_id:
type: stringThe plugin, file, amd worksheet configurations follow the same pattern as the paragraph migration. The difference for files with no header row is reflected in the other parameters. header_row is set to null to indicate the lack of headers and origin is to A1. Because there are no column names to use, you have to use the ones provided by the spreadsheet. In this case, we want to use the first two columns: A and B. Contrary to CSV migrations, the spreadsheet plugin does not allow you to define aliases for unnamed columns. That means that you would have to use A, B in the process section to refer to these columns.
row_index_column is set to null because it will not be used. And finally, in the keys section, we use the A column as the primary key. This might seem like an odd choice. Why use that value if you could use the row_index_column as the unique identifier for each row? If this were an isolated migration, that would be a valid option. But this migration is referenced from the node migration explained in the previous example. The lookup is made based on the values stored in the A column. If we used the index of the row as the unique identifier, we would have to update the other migration or the lookup would fail. In many cases, that is not feasible nor desirable.
Except for the name of the columns, the rest of the migration is almost identical to the CSV example. Small changes were made to prevent machine name conflicts with other examples in the demo repository. For reference, the following snippet shows part of the process and destination section for the Microsoft Excel image migration.
process:
psf_destination_filename:
plugin: callback
callable: basename
source: B # This is the photo URL column.
destination:
plugin: 'entity:file'Refer to this entry to know how to run migrations that depend on others. In this case, you can execute them all by running: drush migrate:import --tag='UD Sheets Source'. And that is how you can use Microsoft Excel and LibreOffice Calc files as the source of your migrations. This example is very interesting because each of the migration uses a different source type. The node migration explained in the previous post uses a Google Sheet. This is a great example of how powerful and flexible the Migrate API is.
What did you learn in today’s blog post? Have you migrated from Microsoft Excel and LibreOffice Calc files before? If so, what challenges have you found? Did you know the source plugin configuration is not dependent on the file type? Share your answers in the comments. Also, I would be grateful if you shared this blog post with others.
Next: Defining Drupal migrations as configuration entities with the Migrate Plus module
This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors. Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.
Image credit: "Anti-Muslim Ban Protests" by Mrs. Gemstone is licensed under CC BY-SA 2.0.
Stay in touch
Drutopia Features
Actions
An action is a call for concrete work from supporters. It can stand on its own or be part of a campaign. It can have a goal and due date set. For example, a call for 50 people to call a prison by June 13th demanding an investigation into prisoner abuse. An action can also be turned into a fundraiser, allowing people to donate directly to the site through Stripe. No longer will groups be dependent on proprietary platforms such as GoFundMe and IndieGoGo.
Articles
Drutopia provides an article content type and related configuration, including article types (such as "Victories" or "Press releases") and topics which span all kinds of content on the site (such as "Tenant organizing" or "Environmental racism").
Blog Posts
Drutopia provides the blog content type and related configuration, ideal for giving individuals or groups more informal voices distinct from the organization as a whole.
Campaigns
A campaign page is a central place to explain an issue, publish updates about its activity, list out demands, post calls to action, and raise funds.
Groups
- Groups broadcast to supporters their activity and ways to support.
- Each group has a page within the website. A group can launch a campaign, call for an action and publish a news article.
- All of these pieces of content are displayed on a group's page, creating a mini-site.
Profile Pages
Drutopia provides a People content type for showing visitors information about staff, volunteers, contributors, and more.
The Drutopia Initiative
The Drutopia initiative lays out a technique for configuring a Drupal distribution, for the commons. It is a recognition and a reflection in software architecture that liberation requires organizing and collective action. We ask you not to sit back and let Drutopia work its magic, but rather to stand up and, with us, build a cooperative of grassroots organizations that work together to build tools that we can own together.
Core values of the Drutopia initiative:
- Be inclusive regarding gender, gender identity, sexual orientation, ethnicity, ability, age, religion, geography, and class.
- Commit to protection of personal information and privacy and freedom from surveillance
- Value collaboration and cooperation above competition
- Prioritize human needs over private profit
- Foster non-hierarchical structures and collective decision-making
Drutopia encourages:
- Users and adopters to pool their resources and take a lead role in prioritizing and crowdfunding new development
- Designers, site builders, and developers to focus on rolling their work into shared solutions available to everyone, heightening the ability of groups of all stripes and sizes to benefit.
Strategy - Shared Configuration Management
Drutopia focuses on a specific use case for websites — grass-roots organizing — and pioneers an elegant model for reducing the costs of maintaining an individual Drupal site by providing a shared base configuration for a collection of websites that are run by various organizations with similar needs that contribute to the development cooperatively.
Learn more specifics about the cooperative development strategy behind Drutopia by reading the Drutopia White Paper.
Drutopia as a LibreSaaS or Platform Cooperative
We believe that the platforms that influence and affect so much of our lives (think Facebook and Google) should be owned by the people who use them. In the same line of thinking, the potential of the Drutopia configuration framework is truly unleashed as fully hosted, member-owned platforms. Members of the platform cooperatives will drive forward the vision of the project from the perspective of those most affected– shaping the development of new features. The platforms themselves can span multiple participating hosts that endorse and follow the hosted Drutopia standards.
Contribute to the Drutopia Initiative
Drutopia is a fully functional content management system apart from all these goals, and there are a few ways different individuals and organizations can contribute to this initiative:
1) Drutopia as a Drupal distribution that can be self-hosted by anyone, but the software project will be democratically governed by members of the Drutopia cooperative. For as little as $10 a year you can become a member who votes in leadership team elections, suggests features for the development roadmap, and is part of a community that is building tools for the grassroots.
2) Non-profits and other organizations that pay Chocolate Lilly or Agaric to build a Drutopia website for their organizing will be contributing to our development efforts, and helping us to move forward in attaining the goals of the Drutopia initiative.
3) Individuals and groups that recognize needs for specialized Drupal distributions should talk to us. We encourage you to reach out and bring together others that share the same needs, so that we can collectivize the development and maintenance of more Drutopia distributions.
Cooperative Webhosting
Staying true to the core values of Drutopia, we hosting your site through, and include membership in, democratically governed May First Movement Technology cooperative. May First is dedicated to supporting organizers and activists by providing tools and services that protect their data and privacy from governments and corporations.
