TBWikiToDo 

TB Wiki

Login

navigation
search
toolbox
  article   edit this page   history  

TBWikiToDo

This file contains notes about the status and progress of TBWiki
Contents:

Note that some test materials are on a separate wiki at: http://bird.org/cgi-bin/test.cgi/FrontPage

Before working on this wiki any more, re-read this article by a developer who has written 7 personal wikis:

Next major items to work on [edit section]

  • MacroCategory.py - show a list of pages in a particular category
    • implement category tags, like this: 
       {{Category(Tasklist,2023)}}
 {{Category(New Feature)}}
    • implement category reference, like this: 
       {{ShowCategory(Tasklist)}}
or
 {{Category(show:Tasklist)}}

  • Blog processor
    • support adding a blog entry
  • (done) ExportSite processor
  • Search macro: 
     {{Search(word or phrase)}}

  • do performance enhancements, possibly using the Scalene profiler
  • add a "blog" processor
  • figure out why I keep getting logged out (is there something wrong with the cookie expiration date?)
  • don't add an extra line when saving.
    • use splitlines instead of split to avoid adding an extra line on an edit??
      • there appears to be no split here
    • do some testing to see where the extra \n gets introduced:
      • client side or server side?
      • in form population (send from server) or in form response data (sent from client)?
    • is there a way to detect a new newline, and remove it?
    • grep split tbwiki_engine.py | grep [\\]n
  • get css from page area of local wiki
    • don't retrieve from /var/www/css
  • Support wiki on mobile devices
    • (done) fix up mobile theme
      • change mobile.css and mobile.footer and mobile.header
      • some pages still show very small fonts on my phone
    • autodetect mobile phone browser, and use mobile theme automatically
      • user UserAgent field
  • switch to wsgi - I considered flask, see FlaskResearch
    • should switch to wsgi - I have experience with that
  • enhance blog system (see TBWikiToDo#Blog)
  • add page lock (to support multi-user)
  • add help routine to all processors
    • add SystemInfo processor to handle help requests
      • with help action item, that calls help with a processor name
      • processors only have to return a block of tbwiki markup
  • make tables robust
    • eliminate save_table_data - placing support in appropriate routines in the io_handler (tb.handler)
      • one of write() or some other class method.
    • fix bug 017. See BugList
    • fix bug with edit-in-place on QuarterlyData table
    • make automated test suite for testing page rendering
    • fix all known bugs, with all known table types
      • make a table test matrix
  • table flow control
    • for transcription program (ProcessorTranscriber.py)
      • make a view form and edit form for the transcriber database
        • read documentation on view and edit forms
      • add processor_name to the transcriber database config
  • add support for hybrid tables
    • use case is to add a spreadsheet formula to the FIXTHIS pages, to show totals
    • ability to have tables with records from different sources
      • add io_handler per-record
      • need permission checks (readable, writable) per-record

Things to do (random ideas, un-triaged) [edit section]

  • don't allow caching a page edit page
    • use http meta cache-control directive to prevent caching
    • set caching relatively low anyway
  • add page locking for edits - to prevent collisions
    • lock a page for 15 minutes when an edit is started
      • create a lock file in the admin directory
        • put user, start time, and page hash in lock file
      • start a timer to handle lock timeout
    • generate a hash of the page when the edit starts
    • if someone else tries to edit, give them a warning that they have lower priority - indicate who has the lock
    • when someone submits a page, compare the page hash and lock
      • if page hash doesn't match, don't save the data
      • remove the lock
    • after 15 minutes, remove the lock
  • support arbitrary field addressability in json data:
    • JsonPageFile:field.list_field[n].foo.bar
  • fix bug: edit-in-place wipes out conf entries in inline block

  • add 'tbclient list' feature
    • modify tbclient to send 'list' request
    • modify ProcessorDataHandler.py to respond to list request
  • add TODAY() function to spreadsheet

  • add websed over snapshotted values
    • add websed over diff of snapshotted values
  • add chart over snapshotted values
    • use plot charts?
  • add access-controlled content
    • the purpose of this is to add a page with contact information for other test projects, on the fuego wiki
    • see https://moinmo.in/HelpOnAccessControlLists
    • at the block level, with ProcessorAccessControl
    • at the page level, with accesscontrol directive
    • put required permissions at top of block
    • prevent reading or writing to block (or page)

  • add link to page for file column in attrdb and parsed data files (to go to page)
    • This is handy for FIXTHIS entries, and for editing
  • _bug0013 - can't edit-in-place the buglist

Major Features that are not complete enough to use [edit section]

User accounts [edit section]

  • Creating and administering user accounts
    • need to re-test user accounts, with new features (next_num)
    • Users can create accounts, but physical access to the server is required to change admin flags, or to remove an account
      • may be able to use a special version of attrdb (raw page edit) for user accounts. It wouldn't be great but would get the job done for now.
      • is the user edit form not working??

Table usability [edit section]

  • Specific table usability features:
    • other usability feature is deciding which links to use for a table (I never use query, and always have to decide between 'edit-in-place' or 'edit')

BugList, FIXTHIS, Regressions [edit section]

  • See the BugList page for bugs that need to be worked on.
    • that bug list is from a database. Here is one that just text on a page:
  • See the CodeFixthisList page for a list of FIXTHIS items in the code
  • See the PageFixthisList page for a list of FIXTHIS items in the pages

Table of combined issues [edit section]

Date  ^ bug_count  ^ FIXTHIS_count  ^ missing_FEATURES  ^ total  ^
2006-10-06 8 37 7 52
2006-08-25 6 35 5 46
2008-10-06 17 78 1 96
2011-04-21 17 78 1 96
2007-07-25 9 62 3? 74
2008-02-04 13 66 (55 cgi-bin+ 11 plugin) 3 82
2017-03-08 15 xx xx 15 + xx + xx
2016-08-02 13 92 3 108
2016-08-12 14 94 2 110
2006-11-15 8 35 6 49
2007-02-06 9 41 5 55
Query
missing column 'OK' in table

Regression count [edit section]

Periodically, check the regression test page to see if things still work as expected.

Here's a table, manually maintained, with regression testing history.
date  ^ version  ^ commit  ^ OK  ^ missing output  ^ bad  ^ bad page list  ^
2016-07-18 1.5.3 61eb2f1e 39 31 4 TestMarkupVariablesStatic, TestSandbox, tbwiki.footer, tbwiki.header
2016-08-12 1.6.2 a45cce77 91 2 2 TestMarkupVariables, TestMarkupVariablesDyamicTest
Query

Other items to work on, by category [edit section]

Style [edit section]

  • enhance themes feature
    • see DocThemes
    • calculate theme from user account or other variable
      • current use of page name is lame
    • allow configurable navigation links
    • allow configurable toolbox links
    • read page dressing from nested pages??
      • allow header and footer pages to be HTML or TBWIKI format
        • if TBWIKI, allow for sub-page references that are loaded, parsed and formatted recursively
  • tweak mobile.css
  • switch to MediaWiki page syntax?? (not yet)
  • have css come from page area??? (would allow editing css using wiki)

General [edit section]

  • convert upload to be a processor
    • allows putting upload on any page
    • control number of files via config
    • can specify link(" action=upload&dest_filename=foo") for missing files - to prompt user if file is missing
    • upload result should be shown in page message?
  • add page rename
    • can do create, move content, delete, but that's awkward
  • generate tbwiki stats from log
  • cache page content, for static content
    • change all prints to output routines (to control routing of output)
  • allow administrators to view and edit files in the 'admin' directory using normal tbwiki access (just like regular pages)
    • treat it like a MoinMoin overlay directory
  • make a mobile viewer
  • add ability to grab content at specific intervals
    • kind of a combination of crontab and websed
    • allows for charting something automatically over time
    • make a crontab processor
      • present a form for when to run, and action
  • make a syntax for block names that are not included in block output
    • we have anonymous blocks, hidden blocks, and named blocks
      • named blocks show both the name and the content. It would be nice to hide the name
    • possibly: #.BlockName ??
  • make tbwiki more relocatable (for use on laptop)
    • on bird.org:
      • move *.cgi scripts from www/cgi-bin to ~/work/tbwiki/cgi-bin
      • put link in www/cgi-bin
      • add base_dir to *.cgi scripts
    • write script for laptop that does:
      • downloads latest tbwiki and tbwiki-data
      • adjust base_dir in *.cgi files
      • adjust base_url in *.cgi files
      • adjust plugin symlinks in tbwiki-data dirs
  • make unit test system - see Unit Test Notes
    • NOTE - already have ProcessorRegressionCheck.py
    • have a page specify a test
    • have markup, and html and compare them
      • here's one unit test: 
               record=bold 1
       markup='''bold'''
       html=<b>bold</b>
    • make a ProcessorUnitTest.py processor
      • read all "UnitTest.*" pages, convert markup to html, compare to html
      • report results
  • put this categorized todo list in a table?
  • change main() in tbwiki_engine.py
    • call main when tbwiki_engine.py is called from command line
    • make a tbwiki.py command?
      • program should have the ability to create a static page from the command line
        • e.g. tbwiki.py FrontPage -o FrontPage.html
      • program should have the ability to run unit tests?

Security [edit section]

  • check permissions on table write operations
  • allow administrators to view and edit files in the 'admin' directory using normal tbwiki access (just like regular pages)
    • treat it like a MoinMoin overlay directory
  • have rights affect the displayed table edit links

Processors [edit section]

  • add help routines for all Processors
  • add a processor for "select an item"
    • used for choosing a group of words to practice
    • used for choosing a story to read in scriptures
    • present a list of items from a database, that are unselected
      • allow a user to select one
    • or, select an item randomly from a list of database records
      • allow user to select it or keep choosing a different one
    • have 3 categories: unselected, in-progress, finished
      • database must "state" field?
    • processor allows only legal state transitions
      • specify legal state transitions (or is this too meta?)
  • support mapping routines (e.g. to convert chapters to pages and back)
  • ProcessorScriptureLookup.py - maybe should be a macro?
    • take a reference, and replace it with the text for the versus
    • this should be very handy for church talks, lessons, etc.

Blog [edit section]

  • add right sidebar with:
    • recent posts
      • categories
      • search bar
      • tags
  • use individual pages for each post
    • create add-post form
      • page schema has title, content, date, tags, id
  • navigation to previous and next posts
    • response area
      • show threaded responses with "reply" button on each one
      • response form with "post comment button
  • show date, tags, response count
  • see example: https://lava-block.com/godot-engine-tutorial-making-a-2d-game-ui-mechanics-sound/
  • support customizable background image
  • ability to drag and drop images to upload to wiki

ProgressChart [edit section]

  • add support for goal lines to the ProgressChart processor
  • handle chart attributes
    • make chart_attrs_class, to generate defaults when attrs are missing
    • create flat namespace for chart attributes
    • translate chart atttributes into call arguments during chart creation
  • un-specialize pychartwrapper.py
    • chart another data type using same wrapper??
      • generic multi-series data chart
      • scripture reading chart?
      • weight chart
      • running chart??
  • support data entry and modification for TimsRunning data
    • see ProcessorProgressChart.py
    • use Add Form for table??
      • support current date for date entry??

SlideShow [edit section]

  • SlideShow
    • make sure all nested parsing works
    • add sidebar list of slides (section 1 headings), with links to pages
    • support show attributes and per-slide attributes (bgcolor, bgimage, margins, etc.)

ImageGallery [edit section]

  • ImageGallery
    • support links=1 - to show individual images in full screen
    • autodetect image size for full-screen (standalone) image gallery
      • use Javascript to detect browser window size??
    • support num_images - to show only some of the images
      • gallery has prev and next links on sides (similar to blog navigation)
    • support image borders in ProcessorImageGallery
    • support link to fullsize image in ProcessorImageGallery
    • support displaying the filenames in ProcessorImageGallery
    • add javascript zoom functions

Tables [edit section]

  • add sortby=select_order - to specify that text fields should be sorted in the order that they appear inm the select possible_values list.
    • this helps the user control the sort order for things like word-based priority: ex: very high, high, medium, low, very low
  • add messages to table module
    • allow for outputting messages about an individual table
    • table.add_table_message(msg)
    • this would be handy for table-specific error messages
    • should have block-specific messages in general
  • allow for configuration of table control links
    • have user specify which links to show, and where
  • use better filenames
    • use unique part of filename for record_id
    • allow user to pick filename when creating the record
  • support edit-in-place for tbwikidb files
    • use get_section and set_section
      • modify to read/write definition lines
  • eliminate difference between record, record_id
  • use tbwikidb or attrdb for user database??
    • this puts human readable filler on every page, but that's the wiki way
    • this allows creating a multi-file database just by creating the template
    • this is fragile - human editors will break the pages parseability
  • add table query cheat sheet to query table form page
  • make prototype family info database on bird.org
    • automatically create a User account file on login???
    • make MemberXX records for family members
      • determine the schema (what fields)
      • create a table view with contact information
  • table queries and views FEATURE
    • support <table>RecordView
  • support read-only attribute on whole table
  • Support time-sampled database
    • schedule a websed operation that grabs data at crontab-like intervals
      • or that creates a new record whenever the page is accessed (and schedule something to access the page automatically at periodic intervals)
  • support automatic data snapshotting
  • write a 'table creation wizard' processor
    • ask for field names and types, and create the schema file and cover file (or template file, if its a tbwikidb)
  • document table config concepts:
    • base config (from original conf block)
    • dynamic config (from request form variables)
    • query config (from query conf block)
  • saved queries
    • specify query list in conf block
    • each item in list specifies a query conf block
  • make backup file when editing table files
  • finish all routines for editing a table
    • use {{{Table(arg1,arg2,arg3...) notation for inline tables
      • support other args:
        • formspec, query line (row filter, column filter, etc.)
  • document Table use with data_scan module
    • DocTables has some information (one example, the FIXTHIS table)
  • multi-file tables
    • display multi-line values correctly
    • display '<no-data>' values correctly
    • auto-sense type of each field (text, select, textarea) and use for auto-form generation
  • handle field types with enumerated values (lists and radio buttons)
  • document table architecture and interfaces - see Table Architecture
    • document table attributes
    • document chart attributes
  • make ProcessorTablePick.py
    • See Table Pick Page
    • present simplified list of "unpicked" items
    • allow user to pick one item
    • save picked item with attribute "state" changed to "in-progress"
    • present detailed list of "in-progress" items
    • allow user to mark an item as "done"
      • save done item with attribute "state" changed to "done"
    • whether user wants to have multiple items in "in-progress" state is their choice
  • queries, query_lists

Forms editing and Templates [edit section]

  • use form emitter for user accounts.

Users and attributes [edit section]

  • Support online creation of new users
    • make a "create user" form
    • create a new user account file from a form
  • provide help if user forgets password
    • link to e-mail the password??
    • password hint??
  • create 'user.edit' action, which edits a user account file
  • support different roles (admin vs. non-admin)
  • allow user to edit their attribute values
    • make an "edit user" form
  • REFACTOR - use existing form code for user forms
  • make user account management it's own processor module
    • ProcessorUser.py - we now have sub-actions, so this should work.
      • it needs to access table operations, but in a special way

Long term ideas [edit section]

  • add 'save expected output and show again' to RegressionTest processor
  • add link to backup page on history page
  • look at other wikis for feature ideas
    • in particular, ikwiki statically compiles the wiki on every page edit
  • documentation
  • table editing:
    • should support "save and edit again" and "save and add another row" buttons for saving edit forms.
  • Table Of Contents
    • do auto-table of contents, like wikipedia??
    • handle included files and text expanded from processors??!!
  • check HTML and CSS on multiple browsers with BrowserCam or browsershots
  • make a journal processor (based on progress chart)
  • make a blog processor (with threaded comments)
  • make a forum processor (with threaded and voted-on comments)
  • make a page cache
    • add dependencies to a makefile, and rebuild cache when dependencies change
  • put config into admin page?
    • reduce cgi-bin to just some directory references

Things Completed [edit section]

  • 2021-03 Remove '?action=save' from url in edit form
  • 2018-10 Add image rotate to ImageGallery app.
  • 2017-04 add triggerd - ability to detect triggers that are due and execute them
  • 2017-03 add ProcessorTrigger - ability to trigger a snapshot
  • 2017-03 add ProcessorSnapshot - page/block/table snapshot function
  • 2017-02 add tbclient - ability to get,put and edit items in the wiki
  • 2017-01 add page trails
  • 2017-01 add relative index to cell address (so you can do $.-1)
  • 2016-12 ProcessorImageGallery - finish showing images in full size page
  • 2106-11 add ability to customize table operation links (edit_links, sort_links)
  • 2016-9 add ability to read a specific block, by number
  • 2016-08 convert to per-table messages (for debugging and errors)
  • 2016-08 support editing sections with nested sub-sections
  • 2016-08 allow default_values with generated values (%(timestamp), %(next_int), %(next_page_num))
  • 2016-08 add page delete
  • 2016-07 add tbwikidb database type
  • 2016-07 fix data_scan module to parse multi-line fields
  • 2016-05 make ProcessorImageGallery
  • 2016-05 make ProcessorBlog
  • 2016-03 make better diff output
  • 2016-03 put engine_version in data_class, and add to footer
  • 2016-03 add "printable format" option for page
  • 2016-03 made a test wiki at: http://bird.org/cgi-bin/test.cgi/FrontPage
  • 2014-05 make talk pages optional - see config.has_talk_page global config
  • 2010-10 support inline tables - FEATURE
  • 2010-?? websed - add websed feature to scrape other web pages
    • 2010-?? data-scan - add as a processor first, then make into a database grabber
  • 2008-10 table - back up page on table edit
  • 2008-09 chart - draw chart of table data
  • 2008-08 processor - add a charting package (pychart)
  • 2008-02 table - added support for sorting table rows
  • 2008-02 table - support col_list and row_filter specifiers
  • 2008-02 table - added ability to edit whole table at a time
  • 2008-02 macro - create Include macro
  • 2008-02 table - separated conf into separate class
    • add field_db (parsing, handling)
      • add field types, colors, possible values
    • add routine to read block from page
  • 2008-02 table data - separated IO handler into separate class
  • move tbwiki with authentication to bird.org
  • aug03 - add user accounts.
    • put user account files in an 'admin' directory
    • authenticate user from files in 'admin' directory in action user.login
    • look up user by name to process login form (in action user.login)
  • jul07 - add cookie code for user accounts
  • feb07 - use pages for page dressing (see tbwiki.header and tbwiki.footer)
  • jan07 - support file upload FEATURE
  • 11/15 - add support for inline images
  • change backup routine to save current version of page instead of old version of file
  • save edit comment and show in history
  • edit sections FEATURE
  • parse and format:
    • bullets
    • headers
    • horizontal rules
    • paragraph breaks
    • preformatting
  • write edit form for page
  • save edited data to file
  • read page from data directory and emit as html
  • write handlers for various actions
  • create add_record form (using existing edit_form code)
  • write mtable.save_form() routine
  • re-write edit_form to present whole table with edit links
  • support Table internal macro, with args
  • modified backup time in file extension to be human-readable
  • wrote mtable.modify_record() routine
  • wrote mtable.remove_record() routine
  • parse and display URLs as links
  • use pages for page dressin
  • make items in table of contents into links
    • make an anchor target for each heading
    • use href for anchor target in TableOfContents macro
  • make nicer format for table of contents box
  • add search form to header
  • support multi-page table source_spec
    • support e.g. "User*" as the spec
      • read all pages starting with User
        • finish parse_table_pages() routine
          • read each page matching the source_spec
          • parse out values using a page template
            • start with simple name=value parser
      • create page on add record
      • write page on edit record
      • remove page on remove record
  • support history action
    • show history from backup times
    • allow diff between two versions

TBWiki engine 1.9.2 by Tim Bird