Let's restructure the question into the two key questions:
- How do we teach intuition or intuitive thinking?
- How do we document experience?
Both are very doable but at different levels of detail.
Intuition is a macro or bigger picture thing which we can really sum up as culture. The need for I.T. to create a culture around security and safety needs to be at the forefront of the company culture. We can teach this through behavior modification techniques:
- Create easy to remember advice / guidelines (e.g. "Too good to be true = it is", "Slow down and think before acting", "Question everything, do not blindly trust anyone", "Ask the I.T. help desk or a manager via a known phone number or email", "Don't click links without carefully inspecting them", "Don't give information out willingly especially personal or private information. When in doubt, it's private and a secret", etc.)
- Create routine internal tests (e.g. Send phishing email tests, Do regular audits, explore physical device security [people leaving devices unlocked], etc.)
- Require routine and continuous training (e.g. self-paced training on I.T. threats, continuous simulations of various incident types and/or events, external testing [hire firms dedicated to testing their ability to breach your security ... keep it on a need to know basis] and auditing, automatic (re)training on people who fail internal I.T. tests, etc.)
- Send routine reminders and examples of "Do not do..." or "Do the following..." which are "tips & tricks" style company wide memos/emails
- Remove individuals from sensitive positions who exhibit problems with maintaining I.T. standards and security
- Conveying a spirit of "You ARE the first line of defense to stopping problems and disasters before they come crisis and failures" with EVERY employee from the CEO to the janitor is of the utmost importance ... but when something does go wrong and that first line of defense fails: having excellent backup systems and the sense "Contact I.T. immediately as time is of the essence in containing and correcting the problem(s)."
- Cultivate a thirst within the team for constant improvement and research of the latest technology trends, cybersecurity threats, and best solutions/strategies for mitigating those threats (e.g. reading credible articles/books/blogs/etc., visiting cybersecurity oriented websites/forums/conferences/etc., and diving into the I.T. world from all angles [esp. ethical hacking ... learning how it's done to prevent it happening in your company]).
Documenting experience is a micro level concern where individual incidents or cases need to be logged in a documentation system (e.g. knowledge base, FAQ, and/or Wiki) that is a S.P.O.T (Single Point Of Truth). Whatever system is used needs to be easy to use both from a "file it" perspective and a "retrieve it" perspective. Good search tools, categorization, and organization are essential to providing a solid documentation system that will be used, maintained, and updated.
Whenever there is a failure or incident, the entire incident must be logged in the system. Everything should be covered from a "Step-by-step breakdown of what happened by all actors" to "What did we do right" / "What did we do wrong" and "How did we fix the problem/enhance the security of the company" point of view. There should be an investigation in all cases that ultimately leads to this documentation getting created and appropriate new policies, procedures, and practices being put into effect immediately.
Establishing standard operating procedures (e.g. checklists and policies) for disasters, breaches, and other I.T. critical events (incl. termination of employees/contractors/etc., transferring users, absorbing new business units/companies, etc.) should also be put in this system (with great detail). All checklists and associated reference materials (e.g. system manuals, accounts, shared credentials, servers, etc.) should be placed here as well. A good book on checklists is "The Checklist Manifesto".
Creating routine processes that update and ensure the information in the system is accurate, current, and clear is just as essential as well (doing this around the holidays when there is a low volume of work is a great idea). Also, creating routine audits of the system are essential to test it functions as intended and is improved in the areas where it is lacking (e.g. information is not easily found or categorized, an incident is not logged, the correct procedures is no used, the system is not consulted for an incident, there are enhancements needed for specific cases). A knowledge base system is only as good as it is current, clear, and correct.
This is the kind of approach used (and even enhanced) in the most successful companies and I.T. focused teams/organizations. Regularly consulting external resources such as Microsoft best practices/feedback, Deloitte/I.T. professional consulting firms, and government agencies feedback/regulations is always a great step ensuring you, your team, your I.T. department, and your company are staying ahead of the latest threats and risks in the I.T. realm.
Best of luck! :)