Stevenson Wrote the book in six weeks in the autumn of 1885, the result, he claimed, of a fevered nightmare he suffered while recuperating from illness. The short book did not appear until January 1886, however, because Stevenson’s wife was shocked by the first version and insisted he burn the manuscript and rewrite it. Very soon after publication it became a critical and popular success, and went on to be a global best-seller. The central idea of the book, that one man may have a ‘Jekyll and Hyde’ personality, a separation of distinct social and moral traits, seemed to say something about the strain between private and public self, personal desire and social duty, that spoke not just to middle-class professional men of late Victorian English society, but to people around the world. The book is a rare instance of the invention of a modern myth – and the story is often known in outline even by those who have never read the book. Indeed ‘Jekyll and Hyde’ has entered the English language as an idiom, used to describe persons or things of a dual character, alternately good and evil. It has of course been staged and filmed many hundreds of times.

Literature Review

Jekyll and Hyde is the story of an eminent London doctor, whose friends worry that he is being blackmailed by a lower-class ruffian, Mr Hyde. These friends soon establish a puzzling association between Dr Jekyll and Mr Hyde, but the doctor will not explain why he indulges his friend. Hyde, who has access to the back door of Jekyll’s substantial mansion and can draw money from his bank account, must have some power over the respectable doctor. Is he being blackmailed by a working-class tough? The professional men who surround Jekyll look on in fear and loathing as the crimes of Mr Hyde escalate – culminating in the murder of an elderly politician, Sir Danvers Carew.

The hunt finds Hyde’s lair in Soho, but the monster has vanished. As Jekyll has hidden himself away in his private laboratory, the locked door is finally forced. They find Jekyll, dead. The last chapter is a written confession revealing all: Jekyll, while investigating the duality of human nature, had discovered a potion that separated good and evil elements in his personality. Thus Dr Jekyll and Mr Hyde were in fact one and the same person! Jekyll had progressively lost the ability to control his experiment, and the dark energy of his creation had gradually overpowered his fragile civilised persona. It is unclear if Jekyll’s last act is to destroy his own monstrous creation, or if this is Hyde’s own act of self-destruction to evade punishment. Radically, it is impossible to say who exactly writes this last confession: ‘He – I cannot say I’, the text stutters.

Narrative style

The story is of a respectable London doctor who separates his primal, antisocial personality traits and embodies them in a smaller, vicious, energetic and animal self called ‘Mr Hyde’ to indulge in questionable pleasures otherwise forbidden by society. The plot, however, the way that Stevenson unfolds this story, is a complex and puzzling relay of different narrators that keeps the reader in suspense. The first readers of the book did not know that Mr Hyde was the same person as Dr Jekyll until the very last chapter, ‘Henry Jekyll’s Full Statement of the Case’. The reader is put in the position of the concerned friends who worry about Dr Jekyll’s nefarious new associate, Mr Hyde.

The double life

The brilliance of the novel is that it can be read as a conservative moral story about the dangers of leading a double life, and yet at the same time it hints at a radically new, modern and dynamic conception of the self beyond simple moral divides. The conservative version appealed to many theologians and moral arbiters of the time, who felt that late Victorian England was entering into a dangerous decadent era. Stevenson, educated in a strict Protestant doctrine, seemed to these readers to be dramatising the danger of indulgence in sinful or excessive behaviour. Like the addict the doctor in many ways resembles, Dr Jekyll’s compulsion becomes steadily worse until it spirals out of control. Priests spoke from the pulpit after the publication of the novel, reading the book as an allegory about the necessity of moral restraint.

Stevenson was certainly fascinated by men who lived double lives. He wrote a play about a notorious instance in Edinburgh – that of a respectable deacon of a church known for his good works who lived a secret life as a gambler, carouser, and bank robber until captured and hanged in 1788. The London literary society of the 1880s and 1890s had a number of men leading scandalous double lives, the most famous being Oscar Wilde, who was eventually arrested and imprisoned in 1895. At the height of its early popularity, Jekyll and Hyde was also a reference point for the notorious ‘Jack the Ripper’ murders in the East End of London. Some suspected that the murderer of the six prostitutes was a doctor conducting perverse acts of revenge. Was Stevenson offering an allegory of the dangers of attempting to separate the good and evil aspects of the soul?

The modern psyche

At the same time, Stevenson’s story of a ‘double personality’ was responding very precisely to advances in modern psychology. In the 1870s and 1880s, French doctors had used the somewhat doubtful device of hypnosis to discover that the same mind could, in some rare cases, seem to split into separate memory chains, or even identities. The case study of a woman called ‘Felida X’ fascinated doctors: her sullen, withdrawn, invalid personality could be switched, when put into a trance, into a more socially outward, amenable self. Over years of study, her doctor claimed that he had switched the places of these personalities. The French called this condition ‘double personality’. This was not a moral matter, a separation of good and evil: it was merely a question of the curious mental dissociation of different personality traits.

An English psychologist called F W H Myers was interested in this unusual phenomenon and in 1885 published an essay on what he termed the ‘Multiplex Personality’. When Myers read Stevenson’s Jekyll and Hyde he immediately wrote to the author to express his enthusiasm for a case study he saw as absolutely continuous with modern psychology. Although Stevenson demurred from saying that he was writing psychology, the novel in places seems to agree with Myers’s assessment. Jekyll writes, very radically, ‘I hazard the guess that man will ultimately be known for a mere polity of multifarious, incongruous and independent denizens’ – that is, as a locus of multiple, different selves. It is a strikingly modern conception of the mind. In many ways, Stevenson’s novel points the way towards a new kind of dynamic psychology, the most influential of which was that formulated by Sigmund Freud in Vienna in the 1880s and 1890s: psychoanalysis. Freud, too, wrote ‘case studies’ of dissociated and doubled selves.

Reference:

https://www.britishlibrary.cn/en/works/jekyllandhyde

The Complete Book Below:

The post Write up on Robert Louis Stevenson’s The Strange Case Of Dr. Jekyll And Mr. Hyde appeared first on DayDreamin’ Comics.

“The Final Problem” is a short story by Sir Arthur Conan Doyle, first published in The Strand Magazine in December 1893. It chronicles the dramatic confrontation between Sherlock Holmes and his arch-nemesis, Professor James Moriarty, culminating in their struggle at the Reichenbach Falls in Switzerland. Doyle conceived the tale as Holmes’s last case, intending to retire the character at the height of his popularity. The story’s shocking conclusion provoked public outcry and inaugurated what readers later called the “Great Hiatus”, which lasted until Holmes’s return in 1903 with “The Adventure of the Empty House.”^[1][2][3]

Chapter 1

Publication History and Context

“The Final Problem” debuted in the December 1893 issue of The Strand Magazine and was subsequently collected in The Memoirs of Sherlock Holmes. Doyle had grown weary of his creation and aspired to devote himself to what he regarded as more serious historical fiction. By eliminating Holmes, he hoped to free his schedule and reputation for other literary pursuits. The move, however, shocked the reading public; many readers reportedly canceled their subscriptions to the magazine, and newspapers published letters of protest. The author himself later acknowledged the ferocity of the backlash in reminiscences and correspondence.^[1][3][4][5]

1.2

Holmes’s Warning and the Shadow of Moriarty

Set in 1891 (though published in 1893), the tale opens with Dr. John H. Watson describing Holmes’s unusual agitation. Holmes arrives at Watson’s practice, urgently requesting refuge. He reveals that he has at last encountered an adversary worthy of his intellect: Professor James Moriarty, a brilliant mathematician turned criminal organizer—“the Napoleon of crime.” Moriarty operates at a remove, weaving a web of agents and intermediaries who insulate him from prosecution. Holmes’s investigations, however, have uncovered enough evidence to bring Moriarty’s network to trial, provided Holmes can survive the retribution he now anticipates.^[1][4]

1.3 Flight Across the Continent

Holmes persuades Watson to accompany him on a sudden departure from London. The pair leave Baker Street and cross to the Continent, eluding multiple attempts on their lives. The pursuit is relentless: at railway stations, on platforms, and through the cities of Europe. Holmes retains his composure, but Watson notes a tension in his friend—a sign of the magnitude of the threat. Ultimately, the fugitives arrive in the Swiss village of Meiringen, planning to continue to Italy after a short rest.^[1]

1.4 The Lure to the Falls

While lodging at a local hotel, Watson is summoned by an urgent note requesting medical aid for a gravely ill Englishwoman. He rushes off, only to discover that the summons was a ruse—an artful stratagem designed to separate him from Holmes. Realizing the deception, Watson hurries back along the mountain path toward the Reichenbach Falls, a roaring cataract whose spray and precipices unleash both awe and dread.^[1]

1.5 The Last Note and the Struggle

At the cliff’s edge, Watson finds no trace of Holmes save for a walking stick (or alpenstock) and a letter pinned beneath a stone. In this poignant message, Holmes explains that Moriarty intercepted him and insisted on a final reckoning—mind against mind, man against man—away from the interference of the law. Holmes anticipates the danger but expresses serenity in leaving behind a world where his life’s work has reached completion. Watson reconstructs the scene from footprints and scuffs: two men grappled near the brink; then the tracks end. The inference is terrible and inescapable. Both Holmes and Moriarty have vanished into the maelstrom.^[1][4]

1.6 Aftermath

Watson returns to England a bereaved friend and faithful chronicler. He preserves Holmes’s papers and legacy, writes the narrative as a memorial, and hints that the detective’s success in breaking Moriarty’s organization stands as a final service to society. The story closes on an elegiac note, with Watson inviting readers to remember Holmes at his best—keen, austere, and wholly dedicated to reason and justice.^[1]

1.7 Principal Characters

Sherlock Holmes

Holmes appears at once at the pinnacle of his powers and under mortal threat. His cool logic, mastery of disguise, and encyclopedic knowledge remain intact, but his vulnerability is sharpened by an opponent who understands his methods. The story adds dimensions to Holmes’s character: a willingness to sacrifice himself, a stoic acceptance of fate, and tender consideration for Watson (to whom he writes with affection and clarity).^[1]

Dr. John H. Watson

Watson narrates with restrained grief. His medical training and observational competence ground the tale’s emotional force: he is both witness and interpreter, providing details about footprints, notes, and the landscape that invite the reader to deduce along with him. His devotion to Holmes is evident in his immediate decision to accompany his friend, and in the simple dignity of his closing tribute.^[1][4]

Professor James Moriarty

Moriarty makes his first on-stage appearance in the Holmes canon here. Doyle sketches him as a brilliant mathematician—author of a treatise on the binomial theorem—who leveraged his intellect to mastermind a sophisticated criminal syndicate. He keeps himself aloof from direct action, “stationary” at the center of a vast web, while subordinates carry out operations. This conception created the archetype of the criminal mastermind in modern detective fiction. Though Moriarty’s direct dialogue with Holmes is brief, it crackles with mutual recognition and icy courtesy, suggesting two minds mirroring each other across a moral abyss.^[1][4][6]

1.8 Major Themes and Interpretations

Genius vs. Genius

Doyle frames the conflict as a duel of equals. Where earlier antagonists fell to Holmes’s deductions, Moriarty anticipates and counters them. The symmetry is striking: both men are scholars, ascetics of a sort, and strategists; yet their ethics diverge absolutely. This structural mirroring heightens the drama and permits the story to function as a kind of intellectual tragedy, in which the only plausible end is mutual destruction—or the appearance of it.^[4][6]

Rational Order and the Abyss

The landscape of the Reichenbach Falls functions as more than setting; it is a symbol of chaos at the edge of the rational world. Throughout the Holmes stories, London represents an arena where logic can be applied to tame disorder. In Switzerland, by contrast, nature’s sublime violence threatens to swallow certainty itself. There, at the brink, the champion of reason confronts a force intent on unmaking order. The result is a tableau that fuses detective fiction with the Romantic sublime.^[4]

Friendship and Testimony

Watson’s narrative is a eulogy. His composure underscores the story’s ethics: duty, loyalty, and the act of bearing witness. “The Final Problem” relies on testimonial reliability—we believe Holmes died because Watson, a trusted narrator, believes it. Doyle thereby explores how communities construct truth from memory and evidence, even when direct observation (the fall itself) is lacking.^[1][4]

Fame, Fatigue, and the Author’s Will

The tale also reflects Doyle’s own ambivalence toward his creation. By staging a spectacular exit, he asserts artistic control over a character who had eclipsed his other work. The resulting controversy—public mourning for a fictional detective—became a cultural phenomenon, illustrating the new power of serialized mass-market fiction to shape public sentiment.^[3][5]

1.9 Narrative Technique

Watson’s Limited Point of View

The choice to have Watson absent during the decisive struggle intensifies suspense and enlarges the interpretive space for readers. Watson’s reconstruction from traces—footprints, scuff marks, a pinned letter—mirrors forensic method. The technique is economical: a few physical details invite the same inferential habits that the stories cultivate in their audience. It is forensic minimalism, aligning the reader with Watson’s grief and uncertainty.^[1][4]

Epistolary Insert

Holmes’s letter—calm, graceful, and valedictory—introduces an epistolary element. It grants a glimpse of Holmes’s internal voice, rare in the canon, and lends the narrative a legalistic clarity: the letter functions almost as a deposition from the dead, fixing motive and circumstance in a document Watson can transmit to posterity.^[1]

Compression and Speed

Unlike the leisurely unravelings typical of other cases, “The Final Problem” is driven by chase and imminent peril. The tempo suits a thriller while preserving Holmesian hallmarks—observation, deduction, and verbal dueling. Doyle’s control of pacing ensures that the final encounter feels both inevitable and sudden.^[1][4]

The Great Hiatus and Holmes’s Return

The publication of “The Final Problem” produced a decade-long gap—the Great Hiatus (1893–1903, in publication terms). Within the fiction, the dates differ: Holmes’s fall occurs in 1891, and his return is set in 1894. The detective’s resurrection arrives in “The Adventure of the Empty House” (1903), which explains that Holmes survived Moriarty’s attack by using baritsu (Doyle’s spelling of a then-obscure martial art) to throw Moriarty into the abyss and then went into hiding to dismantle the remainder of the criminal network, including Colonel Sebastian Moran. Doyle, facing persistent public demand (and attractive pay), revived the character to enormous acclaim, inaugurating The Return of Sherlock Holmes (1905).^[2][4][6]

Moriarty’s Legacy

Moriarty’s presence in the canon is brief, yet his influence is immense. He codifies the trope of the shadow genius who organizes crime at scale, a figure echoed in later literature and media. Writers and critics have debated whether Moriarty is primarily a narrative necessity—a nemesis powerful enough to end Holmes—or a fully realized character. Either way, his aura has grown through adaptation, commentary, and the Holmesian “Game,” where enthusiasts treat the canon as historical documents to be harmonized and expanded.^[4][6]

1.9.5 Setting and the Reichenbach Falls

Geography and Symbolism

The Reichenbach Falls, near Meiringen in the Bernese Oberland, are among Switzerland’s most dramatic cascades. In the story they are approached by a narrow path along precipitous cliffs, with booming water and wind-driven spray. Doyle transforms the site into a metaphorical borderland—between life and death, order and entropy. The falls’ physical peril, the uncertain footing, and the roar that drowns out speech contribute to the scene’s stark theatricality.^[1]

Commemoration

Over time the location has become a pilgrimage site for readers. Plaques, statues, and local markers commemorate the fictional event, attesting to the peculiar way literature reshapes real landscapes by overlaying them with imagined history.^[5][6]

Reception and Cultural Impact

The immediate reaction to Holmes’s death was extraordinary. Contemporary reports describe widespread disappointment and, in some quarters, anger. The outcry reshaped Doyle’s career and the magazine’s fortunes, demonstrating the potent bond between serialized narrative and a mass audience. Long-term, the episode has become emblematic of how creators negotiate control over characters who achieve autonomous cultural life. The phrase “killing your hero” is now a shorthand for bold authorial decisions that challenge readers’ expectations.^[3][5][6]

Significance of the Study

Chapter 2- Moriarty’s Character Analysis

Holmes himself once quipped to Watson that if the doctor had a couple of years to spare, he’d recommended spending it in the study of Moriarty. Given that, this attempt of me here is obviously limited. Nevertheless, let’s dig dipper into the myth of the primordial antagonist. Professor James Moriarty appears only twice in the Holmes canon, the first time in Doyle’s famous ‘The Final Problem’. Published in December, 1893 – the story was supposed to be the curtain call for the adventures of fiction’s greatest detective. The fans’ genuine uproar, coupled with Doyle’s own financial troubles, led the author to resurrect the fallen Sherlock from the depths of the Reichenbach Fall. The other person who took the fall alongside Holmes would not return from the dead like his nemesis, but Doyle would go on to write a prequel called ‘The Valley of Fear’ to flesh out the Professor’s character details. Published in 1915, the outing was the final novel by Doyle about the people of 221B, Baker Street – and a prequel to ‘The Final Problem’. An interesting continuity error occurs here. In the novel, Watson gets to know about Moriarty from Holmes ; whereas in the story – which sequentially takes place later in time – the doctor had no idea about the mastermind criminal’s existence.  

Granted, Moriarty was created as a Deux ex machina to end Holmes’ journey. And yet, he remains one of the most fascinating villains of all time. In ‘The Final Problem’, Holmes uses a phrase to describe Moriarty that has now become iconic – “The Napoleon Of Crime”. That’s a poetic way to put who he was – a ruthless conqueror, a master of his profession. “He is a genius, a philosopher, an abstract thinker.  He has a brain of the first order”, says Holmes about Moriarty. These words, remarkably, can be used to describe Holmes himself. And this is where Doyle succeeds. The antagonist mirroring the protagonist has been a tried and tested trope. Doyle runs with the notion and successfully establish it in a span of a short story. Moriarty is everything Sherlock is, but with a key difference. He’s also everything that Holmes chooses not to be.  

Half the evil of the city, Holmes told Watson, is orchestrated by Moriarty. The man is at the top of the food chain, barely getting his own hands dirty. But he’s in control of everything that is happening in his criminal empire. Every crime can be traced back to him, but one just can’t do so. Moriarty, says Holmes, is like a spider – “He sits motionless… in the centre of its web, but that web has a thousand radiations, and he knows well every quiver of each of them.”  

An antagonist who’s matched his intellectual capabilities. That’s an observation Holmes made during the short story. And there’s ample proof to back up the statement. The Professor was a mathematical genius, penning a work on binomial theorem at the age on 21. The work led to Moriarty becoming the Mathematical Chair of one of London’s universities. The prodigy would go on to write the book ‘The Dynamics of an Asteroid’, a masterwork on pure mathematics.

The post Write up on The Final Problem” sherlock holmes by  Sir Arthur Conan Doyle appeared first on DayDreamin’ Comics.

Literature Review

Background of Study

What is the Management System?

 A database-management system (DBMS) is a collection of interrelated data and a set of programs to access those data. This is a collection of related data with an implicit meaning and hence is a database. The collection of data, usually referred to as the database, contains information relevant to an enterprise. The primary goal of a DBMS is to provide a way to store and retrieve database information that is both convenient and efficient. By data, we mean known facts that can be recorded and that have implicit meaning. The management system is important because without the existence of some kind of rules and regulations it is not possible to maintain the database.

 We have to select the particular attributes which should be included in a particular table; the common attributes to create relationship between two tables; if a new record has to be inserted or deleted then which tables should have to be handled etc. These issues must be resolved by having some kind of rules to follow in order to maintain the integrity of the database. Database systems are designed to manage large bodies of information. Management of data involves both defining structures for storage of information and providing mechanisms for the manipulation of information. In addition, the database system must ensure the safety of the information stored, despite system crashes or attempts at unauthorized access. If data are to be shared among several users, the system must avoid possible anomalous results. Because information is so important in most organizations, computer scientists have developed a large body of concepts and techniques for managing data.

Database Management System (DBMS) and Its Applications: A Database management system is a computerized record-keeping system. It is a repository or a container for collection of computerized data files. The overall purpose of DBMS is to allow he users to define, store, retrieve and update the information contained in the database on demand. Information can be anything that is of significance to an individual or organization. Databases touch all aspects of our lives. Some of the major areas of application are as follows:

• Banking
• Airlines
• Universities
• Manufacturing and selling
• Human resources Enterprise Information

◦ Sales: For customer, product, and purchase information.

 ◦ Accounting: For payments, receipts, account balances, assets and other accounting information.

◦ Human resources: For information about employees, salaries, payroll taxes, and benefits, and for generation of paychecks.

◦ Manufacturing: For management of the supply chain and for tracking production of items in factories, inventories of items in warehouses and stores, and orders for items. Online retailers: For sales data noted above plus online order tracking, generation of recommendation lists, and maintenance of online product evaluations.

 Banking and Finance

◦ Banking: For customer information, accounts, loans, and banking transactions.

◦ Credit card transactions: For purchases on credit cards and generation of monthly statements. ◦ Finance: For storing information about holdings, sales, and purchases of financial instruments such as stocks and bonds; also for storing real-time market data to enable online trading by customers and automated trading by the firm.

• Universities: For student information, course registrations, and grades (in addition to standard enterprise information such as human resources and accounting).
• Airlines: For reservations and schedule information. Airlines were among the first to use databases in a geographically distributed manner.
• Telecommunication: For keeping records of calls made, generating monthly bills, maintaining balances on prepaid calling cards, and storing information about the communication networks. Purpose of Database Systems Database systems arose in response to early methods of computerized management of commercial data. As an example of such methods, typical of the 1960s, consider part of a university organization that, among other data, keeps information about all instructors, students, departments, and course offerings. One way to keep the information on a computer is to store it in operating system files. To allow users to manipulate the information, the system has a number of application programs that manipulate the files, including programs to: ü Add new students, instructors, and courses ü Register students for courses and generate class rosters ü Assign grades to students, compute grade point averages (GPA), and generate transcripts

System programmers wrote these application programs to meet the needs of the university. New application programs are added to the system as the need arises. For example, suppose that a university decides to create a new major (say, computer science).As a result, the university creates a new department and creates new permanent files (or adds information to existing files) to record information about all the instructors in the department, students in that major, course offerings, degree requirements, etc. The university may have to write new application programs to deal with rules specific to the new major. New application programs may also have to be written to handle new rules in the university. Thus, as time goes by, the system acquires more files and more application programs. This typical file-processing system is supported by a conventional operating system. The system stores permanent records in various files, and it needs different application programs to extract records from, and add records to, the appropriate files. Before database management systems (DBMSs) were introduced, organizations usually stored information in such systems.

Keeping organizational information in a file processing system has a number of major disadvantages: Data redundancy and inconsistency. Since different programmers create the files and application programs over a long period, the various files are likely to have different structures and the programs may be written in several programming languages. Moreover, the same information may be duplicated in several places (files). For example, if a student has a double major (say, music and mathematics) the address and telephone number of that student may appear in a file that consists of student records of students in the Music department and in a file that consists of student records of students in the Mathematics department. This redundancy leads to higher storage and access cost. In addition, it may lead to data inconsistency; that is, the various copies of the same data may no longer agree. For example, a changed student address may be reflected in the Music department records but not elsewhere in the system. Difficulty in accessing data. Suppose that one of the university clerks needs to find out the names of all students who live within a particular postal-code area.

The clerk asks the data-processing department to generate such a list. Because the designers of the original system did not anticipate this request, there is no application program on hand to meet it. There is, however, an application program to generate the list of all students.

The university clerk has now two choices: either obtain the list of all students and extract the needed information manually or ask a programmer to write the necessary application program. Both alternatives are obviously unsatisfactory. Suppose that such a program is written, and that, several days later, the same clerk needs to trim that list to include only those students who have taken at least 60 credit hours.

As expected, a program to generate such a list does not exist. Again, the clerk has the preceding two options, neither of which is satisfactory. The point here is that conventional file-processing environments do not allow needed data to be retrieved in a convenient and efficient manner. More responsive data-retrieval systems are required for general use. Data isolation. Because data are scattered in various files, and files may be in different formats, writing new application programs to retrieve the appropriate data is difficult. Integrity problems. The data values stored in the database must satisfy certain types of consistency constraints. Suppose the university maintains an account for each department, and records the balance amount in each account.

Suppose also that the university requires that the account balance of a department may never fall below zero. Developers enforce these constraints in the system by adding appropriate code in the various application programs. However, when new constraints are added, it is difficult to change the programs to enforce them. The problem is compounded when constraints involve several data items from different files.

Significance of the Study

Implementation of Atomicity and Durability in DBMS Atomicity and durability are two important concepts in database management systems (DBMS) that ensure consistency and reliability of data. Atomicity: One of the key characteristics of transactions in database management systems (DBMS) is atomicity, which guarantees that every operation within a transaction is handled as a single, indivisible unit of work. Importance:

 A key characteristic of transactions in database management systems is atomicity (DBMS). It makes sure that every action taken as part of a transaction is handled as a single, indivisible item of labor that can either be completed in full or not at all. Even in the case of mistakes, failures, or crashes, atomicity ensures that the database maintains consistency. The following are some of the reasons why atomicity is essential in DBMS: ○ Consistency: Atomicity ensures that the database always remains in a consistent state.

All changes made by a transaction are rolled back if it is interrupted or fails for any other reason, returning the database to its initial state. By doing this, the database’s consistency and data integrity are maintained. ○ Recovery: Atomicity guarantees that, in the event of a system failure or crash, the database can be restored to a consistent state. All changes made by a transaction are undone if it is interrupted or fails, and the database is then reset to its initial state using the undo log. This guarantees that, even in the event of failure, the database may be restored to a consistent condition.

○ Concurrency: Atomicity makes assurance that transactions can run simultaneously without affecting one another. Each transaction is carried out independently of the others, and its modifications are kept separate. This guarantees that numerous users can access the database concurrently without resulting in conflicts or inconsistent data.

 ○ Reliability: Even in the face of mistakes or failures, atomicity makes the guarantee that the database is trustworthy. By ensuring that transactions are atomic, the database remains consistent and reliable, even in the event of system failures, crashes, or errors. Implementation of Atomicity: A number of strategies are used to establish atomicity in DBMS to guarantee that either all operations inside a transaction are correctly done or none of them are executed at all. Techniques to Implement Atomicity in DBMS: Here are some common techniques used to implement atomicity in DBMS:

○ Undo Log: An undo log is a mechanism used to keep track of the changes made by a transaction before it is committed to the database. If a transaction fails, the undo log is used to undo the changes made by the transaction, effectively rolling back the transaction. By doing this, the database is guaranteed to remain in a consistent condition.

○ Redo Log: A redo log is a mechanism used to keep track of the changes made by a transaction after it is committed to the database. If a system failure occurs after a transaction is committed but before its changes are written to disk, the redo log can be used to redo the changes and ensure that the database is consistent.

 ○ Two-Phase Commit: Two-phase commit is a protocol used to ensure that all nodes in a distributed system commit or abort a transaction together. This ensures that the transaction is executed atomically across all nodes and that the database remains consistent across the entire system.

 ○ Locking: Locking is a mechanism used to prevent multiple transactions from accessing the same data concurrently. By ensuring that only one transaction can edit a specific piece of data at once, locking helps to avoid conflicts and maintain the consistency of the database. Durability: One of the key characteristics of transactions in database management systems (DBMS) is durability, which guarantees that changes made by a transaction once it has been committed are permanently kept in the database and will not be lost even in the case of a system failure or catastrophe. Importance: Durability is a critical property of transactions in database management systems (DBMS) that ensures that once a transaction is committed, its changes are permanently stored in the database and will not be lost, even in the event of a system failure or crash. The following are some of the reasons why durability is essential in DBMS:

❖ Data Integrity: Durability ensures that the data in the database remains consistent and accurate, even in the event of a system failure or crash. It guarantees that committed transactions are durable and will be recovered without data loss or corruption.

 ❖ Reliability: Durability guarantees that the database will continue to be dependable despite faults or failures. In the event of system problems, crashes, or failures, the database is kept consistent and trustworthy by making sure that committed transactions are durable.

❖ Recovery: Durability guarantees that, in the event of a system failure or crash, the database can be restored to a consistent state. The database can be restored to a consistent state if a committed transaction is lost due to a system failure or crash since it can be recovered from the redo log or other backup storage.

❖ Availability: Durability ensures that the data in the database is always available for access by users, even in the event of a system failure or crash. It ensures that committed transactions are always retained in the database and are not lost in the event of a system crash.

Recovery and Atomicity in DBMS Introduction Data may be monitored, stored, and changed rapidly and effectively using a DBMS (Database Management System).A database possesses atomicity, consistency, isolation, and durability qualities. The ability of a system to preserve data and changes made to data defines its durability. A database could fail for any of the following reasons:

 ○ System breakdowns occur as a result of hardware or software issues in the system.

○ Transaction failures arise when a certain process dealing with data updates cannot be completed.

○ Disk crashes may occur as a result of the system’s failure to read the disc.

○ Physical damages include issues such as power outages or natural disasters.

○ The data in the database must be recoverable to the state they were in prior to the system failure, even if the database system fails. In such situations, database recovery procedures in DBMS are employed to retrieve the data. The recovery procedures in DBMS ensure the database’s atomicity and durability. If a system crashes in the middle of a transaction and all of its data is lost, it is not regarded as durable. If just a portion of the data is updated during the transaction, it is not considered atomic. Data recovery procedures in DBMS make sure that the data is always recoverable to protect the durability property and that its state is retained to protect the atomic property. The procedures listed below are used to recover data from a DBMS,

○ Recovery based on logs.

○ Recovery through Deferred Update

 ○ Immediate Recovery via Immediate Update The atomicity attribute of DBMS safeguards the data state. If a data modification is performed, the operation must be completed entirely, or the data’s state must be maintained as if the manipulation never occurred. This characteristic may be impacted by DBMS failure brought on by transactions, but DBMS recovery methods will protect it.

Log-Based Recovery

● The log is a sequence of records. Log of each transaction is maintained in some stable storage so that if any failure occurs, then it can be recovered from there.

 ● If any operation is performed on the database, then it will be recorded in the log.

 ● But the process of storing the logs should be done before the actual transaction is applied in the database. There are two approaches to modify the database: 1. Deferred database modification:

● The deferred modification technique occurs if the transaction does not modify the database until it has committed.

 ● In this method, all the logs are created and stored in the stable storage, and the database is updated when a transaction commits. Immediate database modification:

● The Immediate modification technique occurs if database modification occurs while the transaction is still active.

● In this technique, the database is modified immediately after every operation. It follows an actual database modification. Recovery using Log records When the system is crashed, then the system consults the log to find which transactions need to be undone and which need to be redone.

1. If the log contains the record and or , then the Transaction Ti needs to be redone.

2. If log contains record but does not contain the record either or , then the Transaction Ti needs to be undone. ***************************************************************************

Recovery with Concurrent Transactions Concurrency control means that multiple transactions can be executed at the same time and then the interleaved logs occur. But there may be changes in transaction results so maintain the order of execution of those transactions. During recovery, it would be very difficult for the recovery system to backtrack all the logs and then start recovering. Recovery with concurrent transactions can be done in the following four ways.

1. Interaction with concurrency control

 2. Transaction rollback

3. Checkpoints

4. Restart recovery Interaction with concurrency control : In this scheme, the recovery scheme depends greatly on the concurrency control scheme that is used. So, to rollback a failed transaction, we must undo the updates performed by the transaction. Transaction rollback

Database Management Systems, ACID Properties, and their Implications

Overview

• Database: organized collection of data

 • Database Management System (DBMS): a software system that controls the organization, storage, retrieval, security and integrity of data in a database and across multiple databases.

• Database Management System Pipeline:

1. Accept query from user

2. Process query

3. Output result Step 2 – core of DBMS.

• Query Processing Pipeline. Top-to-bottom. 1. Parse SQL into relational algebra tree

2. Figure out the best order of (atomic) operations

3. Figure out the best way to perform each operation 4. Execute each operation

• Query Execution. Each atomic operation have multiple ways of being executed, based on: 1. Size of tables to operate on

2. Availability/desire to use indexes

3. Need/desire to sort data

4. Algorithmic approach (e.g., sorting vs. hashing) Each operation works with data brought to disk. Concerns: 1. Getting data from disk into main memory and back.

The ACID properties, in totality, provide a mechanism to ensure the correctness and consistency of a database in a way such that each transaction is a group of operations that acts as a single unit, produces consistent results, acts in isolation from other operations, and updates that it makes are durably stored.

Advantages of ACID Properties in DBMS:

1. Data Consistency: ACID properties ensure that the data remains consistent and accurate after any transaction execution.

2. Data Integrity: ACID properties maintain the integrity of the data by ensuring that any changes to the database are permanent and cannot be lost.

 3. Concurrency Control: ACID properties help to manage multiple transactions occurring concurrently by preventing interference between them.

4. Recovery: ACID properties ensure that in case of any failure or crash, the system can recover the data up to the point of failure or crash.

Disadvantages of ACID Properties in DBMS:

1. Performance: The ACID properties can cause a performance overhead in the system, as they require additional processing to ensure data consistency and integrity.

 2. Scalability: The ACID properties may cause scalability issues in large distributed systems where multiple transactions occur concurrently.

3. Complexity: Implementing the ACID properties can increase the complexity of the system and require significant expertise and resources. Overall, the advantages of ACID properties in DBMS outweigh the disadvantages. They provide a reliable and consistent approach to data

4. management, ensuring data integrity, accuracy, and reliability. However, in some cases, the overhead of implementing ACID properties can cause performance and scalability issues. Therefore, it’s important to balance the benefits of ACID properties against the specific needs and requirements of the system.

1 2. Getting more than one query/data management operation executed at the same time. This is were we come to transactions. Transaction (externally) Execution of any user program by a DBMS. Transaction (internally) A series of reads and writes. Informally, a transaction is a sequence (set) operations on the database that serves a single purpose and yields a single outcome. Concurrency in DBMS: Modern DBMS must be able to process multiple transactions at the same time. To do this successfully, DBMS must have certain properties. ACID Properties

(A)tomicitiy: The DBMS executes either all actions of the transaction or none. Incomplete transactions should not result in premanent changes in the database.

(C)onsistency: Each transaction run by itself must preserve the consistency of the database. (I)solation: Each transaction should be protected from effects of other transactions being executed alongside. For each pair of transactions Ti and Tj , it appears to Ti that either Tj has finished execution before Ti started, or that Tj has not commenced yet.

Transaction Concept: Transactions are a set of operations used to perform a logical set of work. It is the bundle of all the instructions of a logical operation. A transaction usually means that the data in the database has changed. One of the major uses of DBMS is to protect the user’s data from system failures. It is done by ensuring that all the data is restored to a consistent state when the computer is restarted after a crash. The transaction is any one execution of the user program in a DBMS. One of the important properties of the transaction is that it contains a finite number of steps. Executing the same program multiple times will generate multiple transactions. Example: Consider the following example of transaction operations to be performed to withdraw cash from an ATM vestibule. Steps for ATM Transaction

1. Transaction
2. Start.
3.  Insert your ATM card.
4. 3. Select a language for your transaction.
5. 4. Select the Savings Account option.
6. 5. Enter the amount you want to withdraw.
7.  6. Enter your secret pin.
8. 7. Wait for some time for processing.
9. 8. Collect your Cash.
10. 9. Transaction Completed. A transaction can include the following basic database access operation.

● Read/Access data (R): Accessing the database item from disk (where the database stored data) to memory variable.

● Write/Change data (W): Write the data item from the memory variable to the disk.

 ● Commit: Commit is a transaction control language that is used to permanently save the changes done in a transaction

Example: Transfer of 50₹ from Account A to Account B. Initially A= 500₹, B= 800₹.

 This data is brought to RAM from the Hard Disk. R(A) — 500 // Accessed from RAM. A = A-50 // Deducting 50₹ from A. W(A)–450

 // Updated in RAM. R(B) — 800 // Accessed from RAM. B=B+50 // 50₹ is added to B’s Account. W(B) –850 // Updated in RAM. commit //

 The data in RAM is taken back to the Hard Disk.

Note: The updated value of Account A = 450₹ and Account B = 850₹. All instructions before committing come under a partially committed state and are stored in RAM. When the commit is read the data is fully accepted and is stored on the Hard Disk. If the transaction is failed anywhere before committing we have to go back and start from the beginning. We can’t continue from the same state. This is known as Roll Back.

(D)urability: Once the transaction has successfully completed, its effects must become permanent in the database and shall be able to survive system crashes. Why Atomicity? Transactions are defined as sequences of actions in pursuit of a single goal. As a result of executing the transaction, either the goal is reached and all effects of all actions from the transaction must take hold, or the goal has been abandoned and then no results of any actions should survive.

Why Consistency? This question lies at the core of our course. The short argument for con[sistency is this. Writing software that works with consistent-state databases (as defined by the requirements of the specific application) is easier than writing software that is not allowed to make this assumption. Inconsistent database states that exist for short periods of time might be benign, but any actions performed on the data while the database is in an inconsistent state are dangerous.

 2 Why Isolation? Nothing will screw with the data in the database more than simultaneous changes to the same data from multiple not yet completed transactions. Further actions in such transactions may wind up with incorrect assumptions about the stored data, and may be unable to produce correct actions. Why Durability? We need to make sure that changes to the database state persist once the process of making these changes succeeds. Effect of ACID Properties on DBMS Design. What do we need to do to ensure that a DBMS has ACID properties? For Atomicity. To ensure atomicity DBMS must:

 • know all actions in each transaction;

• have a roll-back mechanism to undo partial effects of aborted transactions. For Consistency. To ensure consistency DBMS must:

 • ensure that all transactions access the correct data at all times; Ensuring consistency is essentially the DB application’s job. For Isolation. To ensure isolation DBMS must:

 • have a mechanism for correctly interleaving actions from multiple transactions (concurrency control mechanism);

• ensure that all transactions access the correct data at all times; For Durability. To ensure durability DBMS must:

• have access to persistent storage, i.e., storage that does not get emptied when the DBMS is powered down;

• carefully and appropriately manage exchange of data between persistent storage and volatile storage (RAM);

• be able to recover from systemwide crashes.

Satisfying ACID Properties: Relational DBMS Organization

• Read database access (see Fig. 1) – Data is stored in records on pages on physical storage devices (e.g., hard disks). – Pages are retrieved from physical storage and loaded into main memory buffer. – Records are retrieved from pages in main memory buffer and passed to the transaction program where they are stored as variables.

• Write database access (see Fig. 1) – Data stored in the variables of the transaction program is written into a page in main memory buffer – Pages of main memory buffer are written to physical storage.

lec01.468.dvi

Atomicity problems. A computer system, like any other device, is subject to failure. In many applications, it is crucial that, if a failure occurs, the data be restored to the consistent state that existed prior to the failure. Consider a program to transfer $500 from the account balance of department A to the account balance of department B. If a system failure occurs during the execution of the program, it is possible that the $500 was removed from the balance of department A but was not credited to the balance of department B, resulting in an inconsistent database state. Clearly, it is essential to database consistency that either both the credit and debit occur, or that neither occur. That is, the funds transfer must be atomic—it must happen in its entirety or not at all. It is difficult to ensure atomicity in a conventional file-processing system.

DATABASE MANAGEMENT SYSTEMS.pdf

1 Transactions

1.1 Introduction In most situations, many users can query & update a database simultaneously, causing concurrency issues. One user can write to the database while another user reads from the same resource. Or both users may try to write to the same resource. We use transactions to address these problems.

 A transaction is a sequence of multiple actions to be executed as a single, logical, atomic unit. From SQL view, a transaction is in the form of:

• Begin transaction

• Sequence of SQL statements

• End transaction

1.2 The Basic Guarantees: ACID Transactions guarantee the ACID properties to avoid the concurrency problems discussed above:

• Atomicity: A transaction ends in two ways: it either commits or aborts. Atomicity means that either all actions in the Xact happen, or none happen

• Consistency: If the DB starts out consistent, it ends up consistent at the end of the Xact

• Isolation: Execution of each Xact is isolated from that of others. In reality, the DBMS will interleave actions of many Xacts and not execute each in the order of one after the other. The DBMS will ensure that each Xact executes as if it ran by itself.

 • Durability: If the COMMIT succeeds, all changes from the transaction persist until overwritten by a later transaction

SQL transactions are critical for ensuring data integrity, consistency, and reliability in relational databases. They allow multiple SQL operations to be grouped into a single logical unit of work, ensuring that either all changes are applied or none are, thereby preventing partial updates and maintaining the database’s structural integrity. Here’s why SQL transactions are indispensable:

Data Integrity

Transactions ensure that data remains consistent by preventing partial updates. If any part of a transaction fails, the entire transaction is rolled back, maintaining the database’s original state.

Reliability

In environments where multiple users interact with the database simultaneously, transactions prevent interference between concurrent operations, ensuring each transaction executes independently.

Error Handling

Transactions provide a robust error-handling mechanism. If an error occurs during a transaction, the ROLLBACK command can undo all changes, preventing data corruption.

ACID Properties of Transactions

SQL transactions adhere to the ACID properties, which are fundamental for ensuring the reliability and consistency of database operations:

Atomicity

Ensures that all operations within a transaction are treated as a single, indivisible unit. If any operation fails, the entire transaction is aborted, and all changes are rolled back.

Consistency

Guarantees that the database remains in a valid state before and after the transaction. This means that transactions must follow the rules defined by the database, such as constraints and triggers.

Isolation

Prevents interference between concurrent transactions, ensuring they execute independently. This property ensures that the outcome of a transaction is not affected by other transactions running at the same time.

Durability

Ensures that once a transaction is committed, its effects are permanent and survive even in the event of a system failure.

These properties make SQL transactions reliable for handling complex operations in multi-user environments.

SQL Transaction Commands

SQL transaction commands are essential for managing and controlling the execution of operations within a database. They ensure data integrity, consistency, and reliability by grouping multiple SQL statements into a single logical unit of work.

These commands allow developers to start, commit, or roll back transactions based on the success or failure of operations. Below is an overview of the key SQL transaction commands, their syntax, and examples.

BEGIN TRANSACTION

Starts a new transaction. All subsequent operations are part of this transaction until a COMMIT or ROLLBACK is executed.

COMMIT

Saves all changes made during the transaction permanently to the database. Once committed, changes cannot be rolled back.

ROLLBACK

Undoes all changes made during the transaction since the last BEGIN TRANSACTION. This command is used when an error occurs or if the transaction needs to be aborted.

SAVEPOINT

Creates a point within a transaction to which you can later roll back. This allows partial rollback of a transaction without affecting earlier changes.

ROLLBACK TO SAVEPOINT

Rolls back the transaction to a specific savepoint without affecting earlier changes.

Example of a SQL Transaction

Bank Transfer Example

Consider transferring $500 from Account A to Account B in a banking system:

Transaction Concept: Transactions are a set of operations used to perform a logical set of work. It is the bundle of all the instructions of a logical operation. A transaction usually means that the data in the database has changed. One of the major uses of DBMS is to protect the user’s data from system failures. It is done by ensuring that all the data is restored to a consistent state when the computer is restarted after a crash. The transaction is any one execution of the user program in a DBMS. One of the important properties of the transaction is that it contains a finite number of steps. Executing the same program multiple times will generate multiple transactions. Example: Consider the following example of transaction operations to be performed to withdraw cash from an ATM vestibule. Steps for ATM Transaction

1. Transaction
2. Start. 2. Insert your ATM card.
3. 3. Select a language for your transaction.
4. 4. Select the Savings Account option.
5. 5. Enter the amount you want to withdraw.
6.  Enter your secret pin.

 7. Wait for some time for processing.

8. Collect your Cash.  

9. Transaction Completed.

A transaction can include the following basic database access operation.

● Read/Access data (R): Accessing the database item from disk (where the database stored data) to memory variable.

● Write/Change data (W): Write the data item from the memory variable to the disk.

 ● Commit: Commit is a transaction control language that is used to permanently save the changes done in a transaction

****************************************************************************** Data Definition Language (DDL) DDL, which is usually part of a DBMS, is used to define and manage all attributes and properties of a database, including row layouts, column definitions, key columns, file locations, and storage strategy.

 DDL statements are used to build and modify the structure of tables and other objects such as views, triggers, stored procedures, and so on. For each object, there are usually

CREATE,

ALTER,

 and DROP

statements (such as, CREATE TABLE, ALTER TABLE, and DROP TABLE). Most DDL statements take the following form: CREATE object _ name

ALTER object _ name

 DROP object _ name In DDL statements,

 object_name

can be a table, view, trigger, stored procedure, and so on.

Data Manipulation Language (DML) DML is used to select, insert, update, or delete data in the objects defined with DDL. All database users can use these commands during the routine operations on a database. The different DML statements are as follows:

SELECT 0Tstatement

INSERT 0Tstatement

UPDATE statement

DELETE statement

Data Manipulation Language DML commands are used to modify the database. It is responsible for all form of changes in the database. The command of DML is not auto committed that means it can’t permanently save all the changes in the database. They can be rollback.

The concept of ACID transactions has been central to the evolution of modern database systems since the late 1970s.

As businesses and research institutions began relying more heavily on digital data, the need for reliable ways to process complex operations across multiple records became clear. Early relational databases such as IBM’s System R and Oracle were among the first to experiment with enforcing stricter guarantees around data integrity and transaction reliability.

In 1983, computer scientist Andreas Reuter and Theo Härder formally defined the four ACID properties—atomicity, consistency, isolation, and durability—in a landmark paper. Their work gave the database community a common language to describe the requirements for reliable database transactions. Each property was intended to address a specific challenge in ensuring that data remained valid and processed reliably, even when failures occurred:

• Atomicity guarantees that multiple operations in a transaction act as a single unit—either all succeed or all fail.
• Consistency ensures that the database remains in a valid state that adheres to defined rules and constraints.
• Isolation prevents concurrent transactions from interfering with each other.
• Durability makes sure that once a transaction commits, the results are permanent, even after system failures.

This framework became the gold standard for relational database management systems (RDBMS), especially as enterprises adopted them for banking, retail, and telecommunications—industries where financial transactions and regulatory compliance demand absolute accuracy.

Through the 1990s and early 2000s, relational databases like Oracle, IBM DB2, and Microsoft SQL Server continued to refine how ACID guarantees were implemented, often using techniques such as write-ahead logging, locks, and multi-version concurrency control.

These methods allowed organizations to process millions of database operations while maintaining strict consistency.

With the rise of NoSQL databases in the 2010s, some questioned whether ACID guarantees could be preserved in systems designed for horizontal scale and high availability. Early NoSQL platforms often relaxed consistency in favor of performance and availability, leading to the assumption that ACID compliance and NoSQL scalability were mutually exclusive.

However, as demand for data integrity in distributed systems grew, vendors like MongoDB proved that it was possible to support multi-document transactions while still scaling across multiple nodes.

Today, ACID transactions remain as relevant as ever. They provide the foundation for maintaining data integrity in industries where errors are unacceptable, from bank account transfers to healthcare records.

At the same time, modern platforms like MongoDB have extended ACID guarantees to flexible, unstructured data and complex operations across distributed clusters. This combination of reliability and scalability ensures that ACID principles, first defined over 40 years ago, continue to drive the future of database management.

What are ACID transactions?

Transactions explained

At their most basic level, database transactions are a group of database read and write operations that have been completed successfully according to DBMS definitions (e.g., defined transaction criteria). There are two main types of transactions:

• Single transactions: A single transaction refers to a series of one or more database operations resulting in one action, completed successfully. Once complete, the transaction is accepted and can be found in a transaction log. A common example of a single transaction is the withdrawal of money from an ATM.
• Multi-transactions: A multi-transaction, sometimes called a distributed transaction, consists of multiple, interdependent transactions spread across different databases and systems (e.g., distributed systems). Records of these transactions are also found in a transaction log. An example of these transactions includes transferring money from one account to another or an employer issuing a new employee a security badge with a photo.

It’s important to note that some transactions are required to adhere to strict standards of data integrity (e.g., data is complete and correct) and data consistency (e.g., the value is the same across all tables/databases). This is often the case where fiduciary responsibility or regulatory compliance is involved.

Examples include commercial banking, investment brokerage, and legal settlements.

In these situations, a standard adherence to DBMS definitions is not enough, and ACID transactions are required.

ACID transactions

ACID is an acronym that stands for atomicity, consistency, isolation, and durability. Together, ACID properties ensure that a set of database operations (grouped together in a transaction) leave the database in a valid state even in the event of unexpected errors.

Further, ACID transactions provide the level of transactional guarantees that many regulatory agencies require.

Below is a general overview of each ACID transaction element, as well as a description of how a NoSQL document database is able to handle that ACID element. For the purposes of this article, MongoDB Atlas will be used.

Atomicity

Atomicity guarantees that all of the commands that make up a transaction are treated as a single unit and either succeed or fail together. This is important in the event of a system failure or power outage, in that if a transaction wasn’t completely processed, it will be discarded and the database maintains its data integrity.

How MongoDB handles atomicity

In MongoDB, a write operation is atomic on the level of a single document, even if the operation modifies multiple embedded documents within a single document.

For situations that require atomicity of reads and writes to multiple documents (in a single collection or multiple collections), MongoDB supports distributed transactions, including transactions on replica sets and sharded clusters.

Consistency

Consistency guarantees that changes made within a transaction are populated across the database system (e.g., nodes) and in alignment with DBMS constraints. If data consistency is going to be negatively impacted by a transaction in an inconsistent state, the entire transaction will fail.

How MongoDB handles consistency

MongoDB gives the flexibility to normalize or duplicate data to optimize applications. If data is duplicated in the schema, the developer must decide how to keep duplicated data consistent across multiple collections. Some applications require duplicated data to be made consistent immediately, whereas other applications can tolerate reading stale data. Examples are illustrated below:

The post Write up on Tech Geek History: Atomicity in databases “SQL Scripting” (2nd Publication Revision) appeared first on DayDreamin’ Comics.

Chapter 1

 1.1 Atari TOS Version 1.0 TOS

Is the operating system designed for the Atari ST and TT series of computers. The Atari ST and TT use a Motorola 68k based CPU (The same CPU used by early Macintoshes) and was first manufactured in 1985.

The operating system consists of two parts. TOS – “The Operating System”. Kind of a customized version of CP/M provided by Digital Research. GEM – “Graphics Environment Manager”. The GUI shell that runs on top of TOS. This shares much in common with the DOS version of GEM.

Both TOS and GEM are stored entirely in the system’s ROM. This is the GEM desktop running on TOS displaying its desktop info screen. The actual resolution of the default display is 320*200, obviously meant for viewing on a TV or composite monitor, but these screen shots have been enlarged to 640*400 for viewability. This is the GEM desktop running on TOS displaying its desktop info screen.

The actual resolution of the default display is 320*200, obviously meant for viewing on a TV or composite monitor, but these screen shots have been enlarged to 640*400 for viewability. This is a blank desktop. Instead of disk icons, there are file drawers.

 The drives / drawers are assigned letters A, B, C and so on, similar to how CP/M and DOS organize drives. Just imagine if IBM had gone with CP/M for the original IBM PC instead of Microsoft DOS.

We might all be running Digital Research GEM 2004 right no The windows in TOS/GEM are Mac-like and can overlap and resize. None of the versions of GEM for Atari TOS were affected by the Apple lawsuit so all of the Atari versions retained these feature The TOS is the native Atari operating system, which comes together with each Atari computer. The first version, 1.0, also known as RAM TOS was loaded from disk, each next version is burned into system ROM at the factory. Each TOS version Atari has developed, apart from the official number, is also known under an unofficial name: TOS 1.0- RAM TOS, since it was loaded from disk and resided in the system RAM.

 TOS 1.02- Blitter TOS was introduced with the Mega ST computer and contained routines supporting the Blitter, a graphic co-processor. TOS 1.04- Rainbow TOS, since it displays a rainbow Atari logo in the info box.

 TOS 1.62- STE TOS was introduced with STE computer series. TOS 2.0x- Mega TOS was developed for Mega STE computer. TOS 3.0x- TT TOS was developed for the Atari TT030. TOS 4.0x- Falcon TOS also known as 3D TOS is present in the Falcon030 computer and provides 3D-looking window elements. TOS 5.0- the real Falcon030 TOS was supposed to become the native system of Falcon computers, but has never been officially released

. The only available version is a beta release known as TOS 4.92. The non-multitasking TOS, to be differentiated from the MultiTOS, is also known as the SingleTOS. System structure

 The TOS consists of four main parts, each one can be replaced by a RAM-based substitute: BIOS, Basic Input/Output System, provides several hardware dependent functions and is basically compatible with PC BIOS.

Additionally, the ST-BIOS contains an enhancement, known as XBIOS(Extended BIOS). The XBIOS functions are strictly platform dependent and aren’t compatible with any other computer, but ST. GEMDOS, is the main file management subsystem. The GEMDOS has been developed by the Digital Research, Inc. and is compatible with DR-DOS. The GEMDOS may be replaced with MiNT kernel in order to use MiNT networking software. VDI, Virtual Device Interface, is a library of hardware dependent graphic functions. Generally, it is responsible for drawing basic screen elements (points, lines, filled and empty objects, fill textures, fonts etc.) and controlling the mouse. The VDI has also an optional, loadable extension called GDOS, mainly responsible for font management. Latest GDOS versions support Bitstream and True Type character sets.

The VDI may be replaced with NVDI, which is much faster and provides some additional functions. AES, Application Environment Manager, is a fully hardware independent window and application manager. The AES and the VDI, both developed by the Digital Research, Inc., are known as GEM, Graphic Environment Manager. The standard AES (version 3.40 in Falcon TOS) may be replaced with the MultiAES (AES 4.10), a part of MultiTOS 1.08 or with the Geneva, which converts the TOS to a co-operative multitasking system. Besides of that, the system ROM contains also the GEM Desktop, i.e. an application program directly responsible for user interaction with the TOS. This structure, in order to keep compatibility with the TOS, is present in some other operating systems. Short characteristicThe main advantage of the TOS is that it is fast, stable and not memory hungry – if you don’t load many TSRs, it gives you about 3,7 MB of TPA on a 4 MB machine, since it doesn’t utilize the system RAM to keep it’s own code into. But even considering that the system resides in ROM, you must prove that the TOS code is generally compact: the latest version, Falcon TOS, fits in a 512k chip together with the desktop and five resource files for five different nationalities. Also notice, that the TOS will work fine even if there is no hard drive connected – you can run the system from a floppy and have a serious chance, that all necessary system files won’t take more than 1% of the capacity on a standard 1,44M diskette.

‘ The main disadvantage of the TOS is that it doesn’t support true multitasking: only one application can be used at once together with up to six so called accessories, which have to be loaded at startup and must be present in the memory all the time the system is running. The accessories may run in parallel with the main application using some sort of cooperative multitasking – the system switches the CPU time between those seven tasks if they interact with the GEM. If the application or one of the accessories executes an operation which runs without any system call, task switching is impossible and the rest is stopped. Additionally, the TOS doesn’t support big logical drives. You can have up to 16 logical drives, 14 of them can be hard disk partitions. Each one is limited to 32766 data clusters, up to 16384 bytes each. It means, that the maximum capacity of a partition is about 512 MB. Summa summarum, the TOS cannot be currently considered as a modern operating system, but one has to remember that the latest version of the system has been released four years ago. That time big hard drives were rather rarely used in home computers and only one of popular operating systems supported true pre-emptive multitasking, namely the AmigaOS. Now the TOS is a bit out of date and has few devotees in the Atari community. Majority has moved to systems mentioned below.

MultiTOSThe multitasking TOS The MultiTOS has been developed by Atari Corp. to be a TOS replacement for those who would like to have a true multitasking system. Unlike the TOS, the MultiTOS is a fully RAM-based system and in fact consist of the MiNT kernel (that is described in one of the further sections) and a new, multitasking version of the AES (MultiAES). Due to this reason, the MultiTOS can theoretically support all of the MiNT possibilities, since in fact it is nothing else than the MiNT configured to be TOS and GEM compatible.

The MultiTOS provides the true, pre-emptive multitasking and supports many additional functions which are needed in such a system, e.g. memory protection, multiple filesystems, startup sequence scripts, standardized system enhacements, easy task selection, window iconification, loadable accessories, unlimited number of concurrent applications and many more. Additionally, it is much more configurable than the TOS and follows the general trend in programming of a modern operating system. Unfortunately, bad financial condition in 1993. has enforced the Atari Corp. to drop all ST-line support. MultiTOS was one of the victims of that decision. The latest official version, 1.04, was not perfect and has been quickly forgotten by the Atari community. Interestingly, the latest unofficial beta, MultiTOS 1.08, was way better and has began a general guideline for programmers developing software and replacement operating systems. In fact, the MultiTOS 1.08, never officially released, never officially supported and used only by a few heroical devotees – has become a standard. MagiCA way to new hardware The MagiC has been introduced to the market about two years ago by a German company called Application Systems Heidelberg, previously known as the developer of a C compiler (Pure C) and has become popular very quickly, breaking the Atari community into Magic’s devotees and enemies

. The MagiC is a modern operating system without any doubt. It is probably also the only and unique complete TOS replacement, which doesn’t use any of the TOS components. It follows the main MultiTOS guidelines (though some of the MultiAES, or even Falcon AES, functions are missing) and, furthermore, tries to enforce a new standard for software developers by supporting many of Digital Research GEM/2 functions (which are missing in the Atari GEM) as well as several own enhancements. Besides of the multitasking (both pre-emptive or co-operative, depending on a user selection), the MagiC supports a new filesystem in order to allow you to use long filenames and really big disks, easy task management, background DMA transfers, fast GEMDOS, fast AES, fast VDI and many more features. Additionally, although it is a RAM-based system, it is compact and not memory hungry – the whole system takes less than 300 kilobytes. Furthermore, it opens a way for the Atari community a way to use GEM applications with newer hardware – since there’s no hope to see new products of the Atari Corp. as the company is dead, the Application Systems Heidelberg released the MagiC versions for Apple computers and PCs. They are claimed to be GEM compatible and are supposed to allow you to use GEM applications on these platforms. MiNTA bridge to Unix The MiNT is a system originally based on the BSD package ported to ST computers by Eric R. Smith. It was supposed to be a backdoor for the Atari community to the Unix world with loads of BSD based network applications. A short (1992-94) romance of the MiNT and Atari Corp., which decided to convert the system to the MultiTOS kernel, has caused a birth of an unique TOS/Unix hybrid, which gives you simultaneous access to both GEM and BSD application libraries. Since the MiNT is the MultiTOS’s kernel, it has kept all the features described above and, if an AES replacement is installed, it can show you a new face of the MultiTOS. Unlike the MultiTOS however, the Unix-like MiNT is based on a different filesystem, which is much faster and more flexible than the TOS FS. Furthermore, thanks to the excellent network support, the MiNT gives you an unique possibility to convert your Atari to a true Internet server, which will still be able to run GEM and TOS applications! Thanks to this feature, the MiNT has a lot of devotees in the Atari community(they’re recently considered as MiNTquisitors) and is the main competitor for ASH’s MagiC.’

Unlike the Linux, the MiNT contents itself with Motorola 68000 without an FPU, but similarly to other Unix systems it requires much more memory to work in a decent setup. A 4 MB is an absolute minimum, if you want to have a multiuser configuration however, and to run GEM applications at the same time, it will require at least 8 MB or more Operating system summaries Sampling Criteria Problem of Content the five years between 1978 and 1983, the videogame industry saw a huge surge in popularity, a halcyon period of soaring profits and new consoles.

The operating system consists of two parts.

TOS – “The Operating System”. Kind of a customized version of CP/M provided by Digital Research.

GEM – “Graphics Environment Manager”. The GUI shell that runs on top of TOS. This shares much in common with the DOS version of GEM.

Both TOS and GEM are stored entirely in the system’s ROM.

This is the GEM desktop running on TOS displaying its desktop info screen.

The actual resolution of the default display is 320*200, obviously meant for viewing on a TV or composite monitor, but these screen shots have been enlarged to 640*400 for viewability.

The development started in 1983. It was marketed into the 1990’s, both as a GUI shell sold as GEM Desktop for operating systems such as CP/M, DOS and FlexOS but also as a shell less graphics library primarily sold on an OEM basis as that could be used to develop and ship graphical applications that worked on the aforementioned OS’s even though they did not have a graphical shell installed.

It is based on the GSX vector graphics interface library that was primarily built on components the company licensed from Graphic Software Systems and development of it was started as soon as GSX had shipped in 1982 although GEM itself was not officially shipped as a desktop retail product until February 1985 (OEM software was sent out in 1984, though). GEM was highly influenced by both the Xerox Star and the original Apple Macintosh and featured a user interface somewhere in between those two, this was not unexpected as GSS Inc. head honcho Lee Lorenzen had worked on the development of the graphics engine and Interpress for the Star at Xerox.

While not a big seller for the company it did manage to outsell Microsoft Windows by a considerable margin up until the release of Windows 3. The best known version of the system was on Atari ST computers where it was used as the GUI component of TOS. A number of European vendors of IBM PC and other Intel based hardware used the system, most notably Amstrad with their PC compatible budget computers and Apricot with their Sirius compatible systems.

There was also a lawsuit brought in by Apple in 1985 that ran into 1986 that stopped USA based computer manufacturers from shipping it with their systems since Apple had threatened to go after any maker that shipped GEM with their system and Digital Research simply did not have the financial wherewithal to offer OEM’s indemnity from Apple’s lawsuit.

Even as if not a big hit it became quite an influential product in the PC world, not the least since Microsoft Windows was developed in response to it, but there was the air of a missed opportunity about the whole project. GEM worked on hardware that was much slower than what the competition was using for their GUIs which was mostly processors like the 68000, Z8000 and 320016 which outran the 4.77MHz in an IBM XT class machine GEM was designed for quite comfortably, but at the same time GEM on an XT was quite reasonable if not exactly fast.

Even though the basic vector drawing engine in GEM was considerably stronger than what Windows had to offer. The Windows version was simply bog slow compared to GEM and almost unbearable on an XT, showing that DRI’s software was much better optimised for low power hardware such as XT class machines than Microsoft’s shell.

GEM simply gave customers a much better experience on the hardware of the time. Also Digital Research had released a GUI word processor and other graphical applications but for some reason did not develop these products any further.

The Apple lawsuit somehow managed to take the wind out of the marketing of GEM and DRI’s CEO Gary Kildall was by that time showing much more interest in his Grolier CD-ROM project than in promoting GEM. Development of the system and related software was cut down considerably in late 1986 although a GEM 3 was eventually released in 1988, but that release had none of the features that had been in shown in the GEM/XM beta of 1986.

Chapter 2  AES fundamentals

The AES(Application Environment Services forms the highest level of GEM. It deals with all those parts of GEM that go above elementary graphic output and input functions. As the AES works exclusively with VDI and GEMDOS calls, it is completely independent of the graphic hardware, of input devices as well as of file-systems.

The AES manages two types of user programs: Normal GEM applications with file extensions ‘.PRG’, ‘.APP’ or ‘.GTP’, and desk accessories with file extensions ‘.ACC’.

Unless you are using a multitasking operating system such as MagiC, MiNT or MultiTOS, the AES can only run one application and six desk accessories at a time. Desk accessories (with an ‘.ACC’ extension) allow quasi-multitasking even with plain TOS: They are usually special GEM programs loaded (wholly or partially) at boot-up from the root directory of the boot drive (normally C:\), which remain in memory and can be called at any time from GEM (and some TOS) programs by clicking on their entry in the first drop/pulldown menu. In other words, desk accessories can be called and used while another application is running and has its window(s) open, even with a single-tasking operating system such as TOS. Note that this is not real multi-tasking, as the main application is suspended while the accessory is executing and only resumes when the accessory is exited.

Unlike applications, desk accessories don’t interact with the user immediately; most just initialize themselves and enter a message loop awaiting an AC_OPEN message. Some wait for timer events or special messages from other applications. Once triggered, they usually open a window where a user may interact with them. Under TOS, accessories should not use a menu bar and should never exit after a menu_register call. Loading of any resources should happen before the accessory calls menu_register, and these resources should be embedded in the desk accessory rather than being post-loaded, as on TOS versions earlier than 2.06 memory allocated to a desk accessory is not freed at a resolution change; thus memory allocated with rsrc_load is lost to the system after a change of resolution with the early TOS’s.

When a desk accessory is closed under TOS, it is sent an AC_CLOSE message by the system. Following this it should perform any required cleanups to release sytem resources and close files opened at AC_OPEN (the AES closes the accessory’s windows automatically). Following this it should reenter the event loop and wait for a later AC_OPEN message.

The following points are covered in this section:

• Accessories
• Bindings of the AES
• The desktop window
• Data exchange via the GEM clipboard
• Messages
• AES object structure
• Quarter-screen buffer
• Rectangle-list of a window
• Screen-manager
• Toolbar support

For the AES too there have been some interesting developments, as various programmers have meanwhile announced their own AES clones; at present one can mention projects such as N.AES and XaAES. Besides constant evolution one may hope also for source texts of these GEM components.

See also: Style guidelines

.2.1 Accessories

• Startup-code for accessories
• Accessories in MagiC

.2.1.1 Startup-code for accessories

To test whether an application was launched as a program or as a desk accessory, one can proceed as follows:

• If the register a0 has the value zero at program startup, then we are dealing with a normal program launch.
• Otherwise we are dealing with a desk accessory, and register a0 contains a pointer to the (incompletely) filled BASEPAGE. The TPA has already been shrunk appropriately (to the sum of basepage size and the length of the three program segments), but a stack still has to be created.

Note: With this information there is no problem in creating the start- up code for a program in such a way that it recognizes automatically how the application was launched, and to continue the initialization appropriately. With most C compilers the external variable _app in the startup code is initialized automatically, which has the value 0 when the application was launched as a desk accessory. This makes it possible to develop applications so that they may be launched either as desk accessories or as normal programs.

See also:

About the AES   Accessories in MagiC   Program launch and TPA

.2.1.2 Accessories in MagiC

Under MagiC, desk accessories are almost equal to programs. Their windows are maintained at program changes. They may have menus and desktop backgrounds, post-load programs, allocate memory, open/close/ delete/copy files etc.

As there is no longer any reason to close windows at program changes, there is no AC_CLOSE message any more. The system does not distinguish desk accessories from programs, apart from the fact that they may not terminate themselves. As under GEM/2, accessories can also deregister themselves in the menu, using the AES call menu_unregister.

In place of accessories, under MagiC it is more sensible to use applications that simply register one menu bar with one menu, and lie in the APP autostart folder. These applications can be loaded when required, and also removed again afterwards.

Note: As of MagiC 4, desk accessories can be loaded also while the system is running (not just at boot-up). Furthermore accessories can be unloaded by clicking on the corresponding accessory entry in the first menu while the [Control] key is held down. One disadvantage is that at present accessories may not perform Pexec with mode 104.

See also:

About the AES   GEM   Startup-code for accessories   shel_write

2.2 The desktop window

Of the available windows, the desktop or background window plays a special role. It has the ID 0, occupies the whole screen area, is always open and also cannot be closed. The working area is the area below the menu bar. Only in this working area can other programs output to the screen or open their own windows.

Normally the working area of the desktop appears as a solid green area (in colour resolutions) or as a grey raster pattern (in monochrome operation). The screen-manager attends to the screen redraws all on its own; with a call of wind_set, other application programs can anchor any other object tree as a background. In that case too the screen-manager looks after any required redraws of sections of the image. Although this possibility is very alluring, there are several reasons that point against the use of the desktop window; the most important:

• Even under a multitasking-capable GEM (MagiC or MultiTOS), there can be only one screen background. This should be reserved for the program that can make the most use of it – as a rule this is the desktop or a desktop replacement such as the Gemini shell, Thing or Jinnee for instance.

To sum up: If possible, the desktop background should not be used in your own programs.

See also: About the AES   wind_set   WF_NEWDESK

2.3 Data exchange via the GEM clipboard

To store files in the clipboard, one should proceed as follows:

• Delete all clipboard files that match the mask ‘scrap.*’ and ‘SCRAP.*’. Note: The mask ‘SCRAP.*’ must be allowed for because old programs knew nothing of alternative and case-sensitive file-systems.
• Save the data to be stored in one or several formats.
• Send the message SC_CHANGED to all applications in the system and SH_WDRAW to the system shell.

The filename is always ‘scrap.’, the extension (suffix) depends on the selected format here; if possible one should always support one of the following standard formats:

In addition one can support one or more of the following formats (the receiver then has the option of using the option with the greatest amount of information):

The receiving program should first check which of the available files contains the most information, and then use this file.

Important: Each of the files in the clipboard contains the same information on principle, just in different formats. The text processor Papyrus, for instance, imports ‘scrap.rtf’ only if its own format ‘scrap.pap’ could not be found.

It should be clear from the above explanation that only one data object (though in different formats) can be present in the clipboard at any time.

Note: A few old programs, such as First Word and First Word Plus, are promiscuous and the clipoards they create automatically are scattered all over the place – usually the directory they are launched from. Some other applications may then use this clipboard rather than the ‘official’ one on the boot drive!

See also: Clipboard functions   scrp_clear   Style guidelines

2.4 The object structure

Although the data structure of the object tree is not a tree in the sense of a binary tree, it nevertheless possesses within a pointer the logical chaining of a tree, with predecessors and successors (generally called ‘parents’ and ‘children’ respectively). The speci fication of parents and children is made via indices to an array.

The tree structure of the individual objects can be illustrated best with a simple example: A menu is composed at first of the menu bar. This in turn contains several title texts. The title texts therefore are contained directly in the menu bar, and are both children of the object ‘menu bar’, so they move on the same hierarchical level. The object menu bar refers with ob_head to the first menu title and with ob_tail to the last menu title. In the first menu title the pointer ob_next serves for addressing the next menu title. Thus the chaining shows the following structure:

     Menu bar:

   +———+———+——–+

   | ob_head | ob_tail | …    |

   |    o    |    o    |        |

   +—-|—-+—-|—-+——–+

        |         +————————-+

        V                                   V

   +———+———+——–+       +———+———+——–+

   | …     | ob_next | …    |  …  | …     | …     | …    |

   |         |    o    |        |       |         |         |        |

   +———+—-|—-+——–+       +———+———+——–+

   1st menu title |                          n-th menu title

                  +—–> 2nd menu title

The actions that may be performed with a given object is specified in ob_flags. The state of an object is held in the entry ob_state. The entry ob_type determines the object type.

For an exact definition some objects need an additional data structure such as TEDINFO or BITBLK. In that case a pointer to this additional structure will be stored in ob_spec.

Summarising again the total setup of the data structure for objects OBJECT:

        +————-+

         |  ob_next    |   Index for the next object

         +————-+

         |  ob_head    |   Index of the first child

         +————-+

         |  ob_tail    |   Index of the last child

         +————-+

         |  ob_type    |   Object type

         +————-+

         |  ob_flags   |   Manipulation flags

         +————-+

         |  ob_state   |   Object status

         +————-+

         |  ob_spec    |   See under object type

         +————-+

         |  ob_x       |   Relative X-coordiante to parent object

         +————-+

         |  ob_y       |   Relative Y-coordinate to parent object

         +————-+

         |  ob_width   |   Width of the object

         +————-+

         |  ob_height  |   Height of the object

         +————-+

See also:

AES object colours   Object types   Manipulation flags   Object status

2.4.1 AES object types

The following types of object are available for selection:

Note: For G_BOX, G_IBOX and G_BOXCHAR, the component ob_spec of the OBJECT structure does not point to another data structure, but contains further information for the appearance of the object. The following apply:

The high byte is used by the AES only for submenus. If the highest bit of ob_type is 0x8000 and the bit SUBMENU in ob_flags is set, then the bits 8..14 specify which submenu is coupled with the menu entry. Hence each application can have a maximum of 128 submenus. MagiC only reads the low byte from ob_type, apart from the submenu handling. TOS reacts cleanly to unknown object types (such as the purely MagiC types G_SWBUTTON etc.), i.e. the objects are not drawn.

See also: Object structure in AES   AES object colours

2.4.2 AES object colours

The following table contains the predefined object colours. Of course particulars depend on the selected screen resolution, as well as any settings made by the user.

Note: These colours also correspond mostly to the icon colours used under Windows and OS/2. With a suitable CPX module one can set the correct RGB values for the frst 16 colours.

See also: Object structure in AES   AES object types

2.4.3 AES object flags

The manipulation flags of an object determine its properties. The following options can be chosen:

See also: Object structure in AES   AES object types

2.4.4 AES object stati

The object status determines how an object will be displayed later on the screen. An object status can be of the following type:

In GEM/5, CROSSED makes the object draw in 3D:

• If an object is both CROSSED and SELECTABLE, then it is drawn as a checkbox.
• If it is CROSSED, SELECTABLE and an RBUTTON, it is drawn as a radio button.
• If it is a button or a box and it is CROSSED, then it is drawn as a raised 3D shape, similar to Motif.
• If a button is CROSSED and DEFAULT, a “Return key” symbol appears on it (rather like NEXTSTEP).
• Boxes and text fields that are CROSSED and CHECKED appear sunken.

GEM/5 can be detected by calling vqt_name for font 1. If nothing is returned, GEM/5 is running.

Recent FreeGEM builds contain a system based on the GEM/5 one, but extended and backwards-compatible. The DRAW3D state is used instead of CROSSED:

• If an object is both DRAW3D and SELECTABLE, then it is drawn as a checkbox.
• If it is DRAW3D, SELECTABLE and an RBUTTON, it is drawn as a radio button.
• If a button is DRAW3D and DEFAULT, a “Return key” symbol will be on it.
• If an object with a 3D border has the WHITEBAK state, then the 3D border will not have a black edge.
• If a radio button or checkbox has the WHITEBAK state, then it will be drawn with a white background rather than in the colour used by 3D objects.

To check for these abilities, use appl_init and check that bit 3 of xbuf.abilities is set.

See also: Object structure in AES   AES object types

2.5 The quarter-screen buffer

The quarter-screen buffer is required by the screen-manager to save the contents of the menu background when drop-down menus drop down. The ‘QSB’ (the usual abbreviation) is also used for the display of alert boxes. Normally its size should depend on the number of colour planes and the size of the system font, but not on the total size of the screen.

A good formula would be:

  500(characters) * space of one character * colour planes

In ‘ST High’ resolution this would give a value of exactly 8000 (i.e. a quarter of the screen memory). Unfortunately in many cases the AES is not so clever. The following table contains a summary of the algorithm used by some GEM versions:

Note: The GEM versions 1.0 and 1.2 (i.e. up to and including TOS Version 1.02) are not prepared by this for colour graphics cards – one of several reasons why even with the use of a special VDI driver under these GEM versions one can not make use of colour graphics cards.

See also: GEM

2.6 The rectangle-list of a window

To overcome the problem of windows that overlap each other, the AES holds for each window a so-called rectangle-list; when a window is partially obscured, GEM divides the visible portion of that window into the least possible number of non-overlapping rectangles, the details of which are then stored in the rectangle-list. Thus the elements of this list form a record of the currently completely visible working area of the corresponding window.

To redraw a window (or its contents) one first inquires with the function wind_get(WF_FIRSTXYWH) for the first rectangle of the abovementioned list. Then one checks whether this rectangle overlaps with the screen area to be redrawn; then and only then one can redraw this area with the use of vs_clip.

This method will be continued with all remaining elements of the rectangle-list, until the height and the width of a rectangle have the value zero.

See also: Clipping   WM_REDRAW   wind_get   wind_update

2.7 The screen-manager

The screen-manager is always active and supervises the position of the mouse pointer when this leaves the working area of the window of other applications. The areas in question are the frames of the windows, the drop-down menus and the menu bar.

When touching the menu area, the screen-manager automatically ensures that the section of the screen occupied by the menu is saved and later restored again (the quarter-screen buffer is used for this).

Manipulation of the window controllers also do not lead to permanent changes of the screen memory; the result of the interaction with the screen-manager are the so-called message events, which inform the relevant application about the user’s actions.

Note: The ID of the screen-manager can, incidentally, be found easily by a call of appl_find(“SCRENMGR”).

See also: About the AES   GEM   Messages

2.8 Toolbar support

From AES version 4.1 onwards the operating system supports so-called toolbars. A toolbar is an OBJECT tree that is positioned below the information-line of a window (and above the working area) which makes it possible to display buttons, icons etc. in a window.

As already known from the window routines, the management of toolbars is shared betwen the AES and the application. Here the AES is responsible for the following actions:

• Adaptation of the X- and Y-coordinates of the toolbar when the window is moved or its size is changed.
• Ensuring that the window is configured to the size required by the window components and the toolbar.
• Adjustment of the toolbar’s width to the width of the window.
• Redraw of the toolbar on receipt of a WM_REDRAW message.
• Sending of AES messages when the user activates an object of the toolbar.

The application, on the other hand, must look after the following:

• Construction of an OBJECT tree for the toolbar (in particulat one has to ensure that all selectable elements of the toolbox have the status TOUCHEXIT).
• Adjustment of the width of a toolbar object if this depends on the width of the window (may be required when changing the size of the window).
• Handling of USERDEF objects.
• Redrawing all objects whose appearance is to be changed. In this case it is imperative that the rectangle-list of the toolbar is inquired for and/or taken into account.
• Problems that arise in connection with the screen resolution have to be solved. Thus, for instance, the height of an icon in the ST Medium resolution can differ from the height of the icon in the TT030 Medium resolution.

For supporting toolbars in your own programs, you should respect the following points:

• Communication with the AES window-manager
• Problems with wind_calc
• Redraw and updating of toolbars
• Toolbar support under XaAES

See also:

WF_TOOLBAR   WF_FTOOLBAR   WF_NTOOLBAR   WM_TOOLBAR   wind_get   wind_set

2.8.1 Redraw and updating of toolbars

For redraws of (parts of) the toolbar, one has to pay respect to the rectangle-list as usual. As the previous wind_get opcodes WF_FIRSTXYWH and WF_NEXTXYWH only respect the working area of a window, however, two new parameters (WF_FTOOLBAR and WF_NTOOLBAR) were introduced, with whose help the rectangle-list of a toolbar can be interrogated.

A redraw of (parts of) the toolbar may be necessary in the following situations:

• The toolbar contains user-defined objects (USERDEF‘s).
• The status of an object in the toolbar was altered. The area to be redrawn here consists of the size of the object plus the space required for special effects (3D, shadowing, outlining etc.).

Redraw is not necessary in the following cases, for instance:

• The relevant window is iconified. The application does not have to take on any management of the toolbar; this is only required at the restoration of the iconification, the so-called uniconify.
• The toolbar present in the window is to be replaced by another one. In this case a call of wind_set with the opcode WF_TOOLBAR and the address of the new OBJECT tree will suffice.
• The toolbar present in the window is to be removed. In this case a call of wind_set with the opcode WF_TOOLBAR and NULL parameters will suffice.

See also: Rectangle-list of a window   Toolbar support

2.8.2 Toolbars and the window-manager

For handling toolbars an application can have recourse to the window- manager of the AES. In detail:

For tacking on a toolbar to a window, it is sufficient to call wind_set(handle, WF_TOOLBAR, …) with the address of the toolbar object tree. If this call is executed while the window is open, then it is itself responsible for the correct calculation of the height of the toolbar.

To exchange a toolbar for another one, one can have recourse to a call of wind_set(handle, WF_TOOLBAR, …) with the address of the new toolbar. If this call is executed while the window is open, then it is itself responsible for the correct calculation of the height of the (new) toolbar.

To remove a toolbar from a window, it is necessary to call wind_set(handle, WF_TOOLBAR, …) with NULL parameters. If this call is executed while the window is open, then it is itself responsible for the correct calculation of the height of the toolbar.

In addition the following points have to be taken into consideration:

• If a window is closed with wind_close, then any toolbar present will not be removed. At a later reopening the toolbar will be in place once more.
• If a window is removed with wind_delete, then its link to a toolbar will be dissolved.
• To be able to recognize mouse-clicks on toolbar objects, these have to possess the status TOUCHEXIT. When such an object is clicked on, the AES creates a WM_TOOLBAR message which is sent to the relevant application.

See also: AES   GEM   Toolbar support

2.8.3 Problems with wind_calc in toolbar windows

When applying the function wind_calc to windows that possess a toolbar there are several problems to be taken into account:

As this function is not passed a window ID (window handle), the desired sizes cannot be calculated correctly when a toolbar is present in the window. The reason for this is that, quite simply, the AES in this case has no information about the toolbar, and specially about its size.

Hence the values returned by wind_calc in such cases have to be further refined by the application. As the program can access the relevant OBJECT tree (and with this also the height of the toolbar), this should present no problems. In detail:

• When ascertaining the border areas of the window, the height of the toolbar must be added to the height returned by the function.
• When ascertaining the working area of the window, the height of the toolbar must be added to the Y-value (couty) returned by the function.

Note: Besides the height of the actual object, the height of the toolbar should include also the space requirement for special effects (3D, shadowing, etc.).

See also: WF_FTOOLBAR   WF_NTOOLBAR   WM_TOOLBAR   objc_sysvar

2.9 AES bindings

The AES is called via a single subprogram that is passed 6 parameters; these are addresses of various arrays that are used for input/output communications. To call an AES function, the following parameter block must be populated with the addresses of the arrays described below:

typedef struct

{

    int16_t *cb_pcontrol;  /* Pointer to control array */

    int16_t *cb_pglobal;   /* Pointer to global array  */

    int16_t *cb_pintin;    /* Pointer to int_in array  */

    int16_t *cb_pintout;   /* Pointer to int_out array */

    int16_t *cb_padrin;    /* Pointer to adr_in array  */

    int16_t *cb_padrout;   /* Pointer to adr_out array */

} AESPB;

The address of this parameter block (which lies on the stack) must be entered in register d1, and subsequently register d0.w must be filled with the magic value 0xc8 (200). With a TRAP #2 system call the AES can then be called directly. For the Pure-Assembler this could look like this, for instance:

               .EXPORT aes         ; Export function

                .CODE               ; Start of the code-segment

        aes:     MOVE.L 4(sp),d1    ; Address of the parameter blocks

                 MOVE.W #200,d0     ; Opcode of the AES

                 TRAP   #2          ; Call GEM

                 RTS                ; And exit

                .END                ; End of the module

There is no clear information available about which registers may be altered. In fact, however, the corresponding routines in ROM save all registers.

Now to the individual arrays. With each field, designated input or output functions can be performed. The following apply:

Warning: If the operating system supports threads, then it is impera tive that a multithread-safe library is used. In particular one must ensure that each thread receives its own global field (see above).

See also: Sample binding   VDI bindings   TOS list

2.9.1 Sample binding for AES functions

The function ‘crys_if’ (crystal interface) looks after the proper filling of the control arrays, and performs the actual AES call. It is passed one WORD parameter in d0 containing the funtion’s opcode minus 10 multiplied by 4 (for faster table indexing); this gives an index into a table in which the values for control[1], control[2] and control[3] are entered for each individual AES function.

AESPB c;

int16_t crys_if (int16_t opcode)

{

        int16_t i, *paesb;

        control[0] = opcode;

        paespb = &ctrl_cnts[ (opcode-10)*3 ];

        for (i = 1; i < 4; i++)

                control[i] = *paespb++;

        aes (c);

        return (int_out[0]);

} /* crys_if */

The table used for this could be built up as follows, for instance:

.GLOBAL ctrl_cnts

.DATA

ctrl_cnts: .dc.b     0,  1,  0       ; appl_init

           .dc.b     2,  1,  1       ; appl_read

           .dc.b     2,  1,  1       ; appl_write

            …

            …

            …

.END

A fuller version is given in The Atari Compendium pp. 6.39-41. Note that the rsrc_gaddr call must be special cased in a library if you want to use the crys_if binding to call the AES.

See also: AES bindings   GEM

3.0 TOS/ GEM

 It was widely called the “Jackintosh,” a nickname combining the name of the Macintosh (they have similar user interfaces) with Jack, for Jack Trammiel, its creator. Despite its promise, early applications for the machine were simply games and programs ported over from lesser machines. Over the years, however, programmers have come to recognize that the ST is not only lightning fast and highly sophisticated, but it is also programmable on several tiers of its operating system. At the heart of the ST is the same famed Motorola 68000 microprocessor that drives the Macintosh, and the Amiga.

 Over the chip is BIOS, XBIOS, and TOS (collectively known as TOS, or the Trammiel Operating System). Above these are the AES and the VDI, the high-level interfaces that make up the GEM interface from Digitial Research. The AES and VDI were the subjects of the first two books in this series. Most programmers prefer to program through GEM. It’s fast and friendly. With a minimum of fuss, it provides the maximum of features, such as reading the mouse position and providing menus and other services. But faster still, and not lagging far behind in ease of use, is TOS. You can use it to provide your own creative uses of graphics, printer functions, and the disk operating system,

TOS

The Operating System

The TOS is the native Atari operating system, which comes together with each Atari computer. The first version, 1.0, also known as RAM TOS was loaded from disk, each next version is burned into system ROM at the factory.

Each TOS version Atari has developed, apart from the official number, is also known under an unofficial name:

• TOS 1.0 – RAM TOS, since it was loaded from disk and resided in the system RAM.
• TOS 1.02 – Blitter TOS was introduced with the Mega ST computer and contained routines supporting the Blitter, a graphic co-processor.
• TOS 1.04 – Rainbow TOS, since it displays a rainbow Atari logo in the info box.
• TOS 1.62 – STE TOS was introduced with STE computer series.
• TOS 2.0x – Mega TOS was developed for Mega STE computer.
• TOS 3.0x – TT TOS was developed for the Atari TT030.
• TOS 4.0x – Falcon TOS also known as 3D TOS is present in the Falcon030 computer and provides 3D-looking window elements.
• TOS 5.0 – the real Falcon030 TOS was supposed to become the native system of Falcon computers, but has never been officially released. The only available version is a beta release known as TOS 4.92.

The non-multitasking TOS, to be differentiated from the MultiTOS, is also known as the SingleTOS.

System structure

The TOS consists of four main parts, each one can be replaced by a RAM-based substitute:

• BIOS, Basic Input/Output System, provides several hardware dependent functions and is basically compatible with PC BIOS. Additionally, the ST-BIOS contains an enhancement, known as XBIOS (Extended BIOS). The XBIOS functions are strictly platform dependent and aren’t compatible with any other computer, but ST.
• GEMDOS, is the main file management subsystem. The GEMDOS has been developed by the Digital Research, Inc. and is compatible with DR-DOS. The GEMDOS may be replaced with MiNT kernel in order to use MiNT networking software.
• VDI, Virtual Device Interface, is a library of hardware dependent graphic functions. Generally, it is responsible for drawing basic screen elements (points, lines, filled and empty objects, fill textures, fonts etc.) and controlling the mouse. The VDI has also an optional, loadable extension called GDOS, mainly responsible for font management. Latest GDOS versions support Bitstream and True Type character sets. The VDI may be replaced with NVDI, which is much faster and provides some additional functions.
• AES, Application Environment Manager, is a fully hardware independent window and application manager. The AES and the VDI, both developed by the Digital Research, Inc., are known as GEM, Graphic Environment Manager. The standard AES (version 3.40 in Falcon TOS) may be replaced with the MultiAES (AES 4.10), a part of MultiTOS 1.08 or with the Geneva, which converts the TOS to a co-operative multitasking system.

Besides of that, the system ROM contains also the GEM Desktop, i.e. an application program directly responsible for user interaction with the TOS.
This structure, in order to keep compatibility with the TOS, is present in some other operating systems.

Short characteristic

The main advantage of the TOS is that it is fast, stable and not memory hungry – if you don’t load many TSRs, it gives you about 3,7 MB of TPA on a 4 MB machine, since it doesn’t utilize the system RAM to keep it’s own code into. But even considering that the system resides in ROM, you must prove that the TOS code is generally compact: the latest version, Falcon TOS, fits in a 512k chip together with the desktop and five resource files for five different nationalities. Also notice, that the TOS will work fine even if there is no hard drive connected – you can run the system from a floppy and have a serious chance, that all necessary system files won’t take more than 1% of the capacity on a standard 1,44M diskette.

The main disadvantage of the TOS is that it doesn’t support true multitasking: only one application can be used at once together with up to six so called accessories, which have to be loaded at startup and must be present in the memory all the time the system is running. The accessories may run in parallel with the main application using some sort of cooperative multitasking – the system switches the CPU time between those seven tasks if they interact with the GEM. If the application or one of the accessories executes an operation which runs without any system call, task switching is impossible and the rest is stopped.

Additionally, the TOS doesn’t support big logical drives. You can have up to 16 logical drives, 14 of them can be hard disk partitions. Each one is limited to 32766 data clusters, up to 16384 bytes each. It means, that the maximum capacity of a partition is about 512 MB.

Summa summarum, the TOS cannot be currently considered as a modern operating system, but one has to remember that the latest version of the system has been released four years ago. That time big hard drives were rather rarely used in home computers and only one of popular operating systems supported true pre-emptive multitasking, namely the AmigaOS. Now the TOS is a bit out of date and has few devotees in the Atari community. Majority has moved to systems mentioned below.

MultiTOS

The multitasking TOS

The MultiTOS has been developed by Atari Corp. to be a TOS replacement for those who would like to have a true multitasking system. Unlike the TOS, the MultiTOS is a fully RAM-based system and in fact consist of the MiNT kernel (that is described in one of the further sections) and a new, multitasking version of the AES (MultiAES). Due to this reason, the MultiTOS can theoretically support all of the MiNT possibilities, since in fact it is nothing else than the MiNT configured to be TOS and GEM compatible.

The MultiTOS provides the true, pre-emptive multitasking and supports many additional functions which are needed in such a system, e.g. memory protection, multiple filesystems, startup sequence scripts, standardized system enhacements, easy task selection, window iconification, loadable accessories, unlimited number of concurrent applications and many more. Additionally, it is much more configurable than the TOS and follows the general trend in programming of a modern operating system.

Unfortunately, bad financial condition in 1993. has enforced the Atari Corp. to drop all ST-line support. MultiTOS was one of the victims of that decision. The latest official version, 1.04, was not perfect and has been quickly forgotten by the Atari community. Interestingly, the latest unofficial beta, MultiTOS 1.08, was way better and has began a general guideline for programmers developing software and replacement operating systems. In fact, the MultiTOS 1.08, never officially released, never officially supported and used only by a few heroical devotees – has become a standard.

MagiC

A way to new hardware

The MagiC has been introduced to the market about two years ago by a German company called Application Systems Heidelberg, previously known as the developer of a C compiler (Pure C) and has become popular very quickly, breaking the Atari community into Magic’s devotees and enemies.

The MagiC is a modern operating system without any doubt. It is probably also the only and unique complete TOS replacement, which doesn’t use any of the TOS components. It follows the main MultiTOS guidelines (though some of the MultiAES, or even Falcon AES, functions are missing) and, furthermore, tries to enforce a new standard for software developers by supporting many of Digital Research GEM/2 functions (which are missing in the Atari GEM) as well as several own enhancements.

Besides of the multitasking (both pre-emptive or co-operative, depending on a user selection), the MagiC supports a new filesystem in order to allow you to use long filenames and really big disks, easy task management, background DMA transfers, fast GEMDOS, fast AES, fast VDI and many more features. Additionally, although it is a RAM-based system, it is compact and not memory hungry – the whole system takes less than 300 kilobytes. Furthermore, it opens a way for the Atari community a way to use GEM applications with newer hardware – since there’s no hope to see new products of the Atari Corp. as the company is dead, the Application Systems Heidelberg released the MagiC versions for Apple computers and PCs. They are claimed to be GEM compatible and are supposed to allow you to use GEM applications on these platforms.

MiNT

A bridge to Unix

The MiNT is a system originally based on the BSD package ported to ST computers by Eric R. Smith. It was supposed to be a backdoor for the Atari community to the Unix world with loads of BSD based network applications. A short (1992-94) romance of the MiNT and Atari Corp., which decided to convert the system to the MultiTOS kernel, has caused a birth of an unique TOS/Unix hybrid, which gives you simultaneous access to both GEM and BSD application libraries.

Since the MiNT is the MultiTOS’s kernel, it has kept all the features described above and, if an AES replacement is installed, it can show you a new face of the MultiTOS. Unlike the MultiTOS however, the Unix-like MiNT is based on a different filesystem, which is much faster and more flexible than the TOS FS. Furthermore, thanks to the excellent network support, the MiNT gives you an unique possibility to convert your Atari to a true Internet server, which will still be able to run GEM and TOS applications! Thanks to this feature, the MiNT has a lot of devotees in the Atari community(they’re recently considered as MiNTquisitors) and is the main competitor for ASH’s MagiC.

Unlike the Linux, the MiNT contents itself with Motorola 68000 without an FPU, but similarly to other Unix systems it requires much more memory to work in a decent setup. A 4 MB is an absolute minimum, if you want to have a multiuser configuration however, and to run GEM applications at the same time, it will require at least 8 MB or more.

To be continued…

Operating system summaries

System component replacements

A computer’s operating system is an organized collection of small built-in programs that enables the computer to communicate with external devices, such as the keyboard, display screen, and disk drive, and to perform fundamental tasks like loading and running an application program. While most people regard GEM as the ST’s operating system, it’s just a friendly user interface on top of a more conventional operating system to make the computer easier to program and to operate.

Chapter 5  What is GEM?

Graphics Environment Manager (GEM) is a graphical user interface created by Digital Research. Originally designed for use on personal computers running the CP/M-80 operating system and later adapted for Atari ST and PCs running DOS. GEM provided a user-friendly interface that allowed users to interact with their computer through graphical icons and windows rather than purely text-based commands. This made computers more accessible to a broader audience, contributing to the rise of personal computing in homes and offices around the world. It provided a windowed environment, allowing users to interact with applications in a more intuitive, visual way, akin to early versions of the Macintosh and Windows operating systems.

How does GEM handles memory management?

Graphics Environment Manager (GEM) handles memory management through a combination of its built-in memory allocation routines and support for multitasking. It uses memory blocks to allocate space for graphical elements, applications, and system resources. GEM ensures efficient use of available memory by managing the allocation and deallocation of these blocks. The system also includes mechanisms to prevent memory leaks and optimize performance. Overall, GEM’s memory management helps maintain system stability and responsiveness while running graphical applications.

Did GEM support color displays?

Yes, GEM did support color displays. Initially designed to run on a variety of hardware, including the monochrome displays of early PCs, GEM was also adapted to utilize the color capabilities of more advanced systems like the Atari ST, which featured a color graphical display. This adaptation allowed GEM to offer a richer visual experience on computers that supported color graphics.

How does GEM compare to other GUI systems from its time?

GEM distinguished itself from other graphical user interfaces of its time through its focus on simplicity and ease of use. It offered a basic graphical environment with straightforward windowing and menu systems. In contrast, contemporaneous GUIs like Windows 3.1 and Macintosh System Software provided more advanced features such as enhanced multitasking and more sophisticated graphical capabilities. GEM’s memory management and resource handling were also less advanced compared to these systems. Overall, while GEM was effective in its niche, it lacked some of the advanced functionalities and performance optimizations found in its competitors.

Does GEM have multitasking capabilities?

GEM, in its original form as developed by Digital Research, did not support true preemptive multitasking. It was primarily designed as a graphical user interface for single-tasking operating systems like CP/M and DOS. However, there were versions and derivatives of GEM, especially on systems like the Atari ST, that implemented cooperative multitasking within applications, albeit with limitations compared to modern multitasking environments.

What are some advantages of using GEM?

Using GEM offers several advantages. It provides a straightforward and user-friendly graphical interface, making computers more accessible to a broad audience. Its window-based system facilitates the management of files and applications. Additionally, GEM is relatively lightweight and requires less memory compared to other graphical user interfaces, which suits the hardware limitations of early personal computers.

Can I customize GEM?

You can personalize your GEM environment by rearranging icons on the desktop, changing desktop wallpaper, and customizing certain settings like screen colors and fonts.

Does GEM have any modern equivalents?

GEM doesn’t have direct modern equivalents since GUI development has significantly evolved. However, its conceptual legacy lives on in modern desktop environments like KDE, GNOME, and Windows Explorer, which offer sophisticated graphical interfaces, multitasking capabilities, and user accessibility. These systems reflect GEM’s early contributions to making computing more visually interactive and user-friendly, albeit with vastly enhanced functionality and complexity.

What was the significance of GEM’s graphical capabilities?

The significance of GEM’s graphical capabilities lies in its pioneering role in bringing GUIs to a wider audience, particularly on PCs where command-line interfaces were standard. By offering an accessible, window-based GUI, GEM made computers easier to use and more visually appealing, paving the way for the widespread adoption of graphical environments in personal computing. Its impact is seen in how we interact with computers today.

Can I run applications within GEM?

Yes, you can run applications within GEM. It was designed as a graphical user interface that allowed users to launch and manage applications through its desktop environment. Applications specifically developed or adapted for GEM could be executed directly from the GEM Desktop, enabling users to access various software tools like word processors, drawing programs, and other utilities within the GEM interface.

How did GEM handle file management?

GEM handled file management through its Desktop, which provided a graphical interface for users to interact with files and folders visually. Users could open, copy, move, and delete files using the mouse, making these tasks more intuitive compared to command-line operations. This approach to file management was innovative at the time, offering a user-friendly way to navigate and organize computer files.

What role did GEM play in the competition between operating systems?

GEM contributed to the competition between operating systems by offering an alternative to the text-based interfaces of DOS. However, it ultimately couldn’t compete with the dominance of Windows in the PC market.

How did GEM handle printing?

GEM handled printing through a straightforward interface that allowed users to print documents directly from within applications. It included a print dialog box where users could select printers, adjust print settings, and initiate the printing process. This integration of printing functionalities made it easier for users to produce hard copies of documents, contributing to GEM’s appeal as a user-friendly graphical environment for early personal computers.

Did GEM have networking capabilities?

GEM itself did not inherently possess networking capabilities as its primary focus was on providing a graphical user interface for single-user computers. Networking and connectivity were not integrated features of GEM. However, other applications and later developments in computing environments could enable network functionalities on systems running GEM, allowing for file sharing and communication over networks through additional software solutions.

How did GEM compare to Windows 3.1?

GEM compared to Windows 3.1 as a more basic and less feature-rich graphical user interface. While both systems used windows and icons, Windows 3.1 offered more advanced features, better multitasking, and improved graphical capabilities. GEM’s simplicity and lightweight design were advantageous for early personal computers, but Windows 3.1 provided a more sophisticated and capable environment.

While GEM provides higher level functions like support for drop-down menus, dialogs, and icons, it still must rely on a set of low-level operating system routines for tasks such as reading a file from the disk drive. It is this set of low-level routines that shall be referred to as TOS. TOS Organization The ST operating system is contained in a set of TOS ROM (Read-Only Memory) chips that contain a total of 192 kilobytes of program code and data.

The name TOS may have at one time stood for The Operating System, but is now more commonly thought of as an acronym for Tramiel Operating System, named after the Tramiel family that now owns Atari. The TOS ROMs contain all of the ST’s system software. This includes: GEM This software provides ST applications with a consistent user interface, featuring drop-down menus, multiple windows, icons, and dialog boxes.

GEM is divided into the VDI (which provides low-level graphics calls), AES (which provides user services like menus), and the GEM Desktop pro

gram, (which provides the desktop metaphor for working with the disk filing system).

 XBIOS XBIOS, the extended Basic Input/Output System is a set of hardware-specific I/O-related routines. There are routines for finding and changing the address of screen display memory, for setting the hardware color registers, for waiting for the vertical blanking interval, and for accessing the sound chip. There are also routines for communicating with the 68901 Multi-Function Peripheral (MFP) chip. GEMDOS (or BDOS) This is a set of functions used to implement the higher level disk filing system, which closely follows the model of MS-DOS. These routines allow the user to access the disk device on the file level, rather than directly reading specific physical sectors on the disk. They allow the user to perform functions like reading the disk directory, creating or deleting a subdirectory, deleting a file, renaming a file, and so on.

The GEMDOS also contains miscellaneous routines fo

computer is first turned on, or the reset button is pushed. It checks for ROM cartridges, configures the I/O ports and the screen, tests memory size, sets the exception handler vectors, executes the programs in the AUTO folder, and jumps to the GEM Desktop program. Since all of the GEM code is included in the ROMs, it will be considered as part of TOS. For purposes of this book, however, TOS will be considered to be everything in the ROMs except GEM. Calling TOS from Machine Language The various TOS routines are called via the exception vectors. When programming in machine language, the general procedure is to push the function number and other function parameters on the stack, issue a TRAP instruction, and then remove the parameters from the stack.

 The specific TRAP instruction depends on the type of TOS routine you’re calling. The BIOS routines are called with a TRAP #13 instruction, the XBIOS with a TRAP #14, and the GEMDOS routines with TRAP #1. A call to the BIOS routine Bconstat() would look like this: move.w # 2 , — (sp) * push device number for console device move.w # 1 , — (sp) * push function number for Bconstat trap #13 * call BIOS addq.l #4,sp * pull the parameters off the stack Note that if you plan to use GEM calls in your machine language application, you’ll need to do some preparatory work at the beginning of the program.

When GEMDOS starts an application program (but not a desk accessory), it allocates all of the system memory to that program. Therefore, if a program uses the system memory-management calls, or any of the GEM AES calls that themselves allocate memory, or runs another program using the Pexec() function, at startup time it must deallocate all of the memory it isn’t actually using. This is done using the GEMDOS Mshrink() function.

Mshrink(). For now, it is sufficient to know that this step is necessary for programs using GEM or memory-management calls, and not for programs that only use TOS function calls. Calling the TOS Routines from C It’s much simpler to call the TOS routines from C than from machine language, since most C compilers for the ST include library routines for the BIOS, XBIOS, and GEMDOS calls. These library routines make calling TOS routines exactly like calling any other kind of C routines.

 For example, the library call bios() issues the TRAP #13 command after the parameters have been pushed on the stack. To call the BIOS routine Bconin() to get a character from the console device (device 2), you need only use the statement: bios(l,2); Many C compilers include a header file called OSBIND.H.

 This header file contains C macro definitions for the various BIOS, XBIOS, and GEMDOS commands. For example, the macro Bconstat(a) is defined as follows: #define Bconstat(a) bios(l,a) Therefore, if you’ve #included the OSBIND.H file in your program, you could replace the bios(l,2); statement with: Bconstat(2); Since this is more readable than the bios() call, the macro format will be used wherever possible. Just remember that in order for the compiler and linker to understand these macros, you must use the #include directive to add the OSBIND.H file first. C programmers usually don’t have to worry about releasing extra memory with the Mshrink() command, or setting the program stack, since this work is done for them by the compiler’s own startup code. This code is found in the GEMSTART.O or APPSTART.O module linked in by Alcyon C programmers, and in the INIT.O module of the SYSLIB library of Megamax C. You should note, however, that in some extreme cases, you may have to recompile the startup module to give back more or less memory than the default modules

The functions that make up the GEMDOS (GEM Disk Operating System) form the highest level of TOS. These functions are also sometimes referred to as the BDOS (using the old CP/M and MS-DOS operating system terminology). They include a wide variety of character device functions, several process control and memory management functions, and a large number of functions used to control file I/O and the filing system. The character device and system routines will be covered in this chapter, while the next chapter will be devoted to the filing system routines. In many cases, GEMDOS functions are modeled after similar commands available under the MS-DOS operating system used on the IBM PC. In fact, most of them share the same function numbers as their DOS counterparts. That’s why GEMDOS functions are often referred to by a hexadecimal function number, just like MS-DOS functions. Like the BIOS and XBIOS functions, GEMDOS routines can be called from user mode.

As with those functions, GEMDOS uses registers A0-A2 and D0-D2 as scratch registers; assume that it changes their contents. If you are programming in machine language and your program uses these registers, save their contents before making a GEMDOS call and restore them after the call terminates. Each of the GEMDOS routines is associated with a command number, and some use command parameters that specify more precisely what they should do. For example, the GEMDOS function to write a character to the console screen has a command number of 2. It requires a single command parameter that tells the function which character to print. To call a GEMDOS function from machine language, push the command parameters onto the stack, followed by the command number, and execute a TRAP #1 statement. The TRAP #1 instruction puts the program into supervisor

mode and begins executing the instructions found at the address stored in exception vector 33, whose address is 132 ($84). This exception vector contains the address of the GEMDOS handler which reads the command number on the top of the stack and directs program execution to the appropriate function. When the function terminates, the program returns to user mode, and the results, if any, are returned in register DO. In most cases, the value is returned as a longword, but there are exceptions. Some error codes are returned as words, so it’s best to test only the low-order words when checking for errors. Also be aware that sometimes a GEMDOS function will return a BIOS error number (between – 1 and -31).

 When a GEMDOS function call is completed, the calling program is responsible for adjusting the stack to remove the command parameters and command number. The following program fragment demonstrates how to print the character A on the console screen using GEMDOS command 2: move.w #’A ‘, — (sp) * push the character value on stack move.w # 2 , — (sp) * push GEMDOS command number on * stack trap # 1 * call GEMDOS handler addq.l #4,sp * pop parameters (4 bytes) off stack Calling the GEMDOS routines from C is much simpler. Most C compilers come with a library routine called gemdos() that stacks the parameters and executes the TRAP #1 instruction. For example, the sample call illustrated above could be accomplished in C by the single statement: gemdos(2, “A’); Since it’s easier to remember a command

name than a command number, most C compilers include a header file called OSBIND.H that defines macros for all of the GEMDOS functions. For example, the macro definition for GEMDOS command 2 is: #define Cconout(a) gemdos(0x2,a) Therefore, after you #include OSBIND.H in your program, you can call your sample function like this: Cconout(‘A’);

Since this format is the more readable of the two, it will be used in the macros in the discussion of GEMDOS routines and sample programs. To use GEMDOS functions in your C programs, link your program with the compiler library that contains the gemdos() function, and #include OSBIND.H if you use the macros.

Character Device I/O Like the BIOS and XBIOS, GEMDOS includes a number of functions that allow you to read characters from a character device, write characters or strings to such devices, and check their input and output status. Tliese functions aren’t implemented in exactly the same way as the BIOS and XBIOS versions, however. Where the BIOS and XBIOS tend to use one function for many devices and require a device number as an input parameter, GEMDOS provides a separate function for each device. GEMDOS also throws in a number of variations. Some keyboard input functions echo the character to the screen, some don’t. Some wait for a keypress to return, others return immediately. There are three functions that wait for a character to be entered at the console keyboard and return the value of the character that was entered. The macro for the first of these functions, Cconin(), is called like this: long keycode; keycode = Cconin();

When called, this function doesn’t return until a key is pressed, unless a keypress is already waiting in the queue. The keycode the function returns is a longword containing both the ASCII value of the key(s) pressed and the scan code. The ASCII value is returned in the low byte of the low word of keycode, while the scan code is returned in the low byte of the high word. See Appendix J for a complete list of keycodes. This call is much like the BIOS function Bconin(), except that Cconin() sends a copy of the key that was pressed out to the console screen. The other two character input functions, Crawcin() and Cnecin(), don’t echo the key that is pressed. These functions are called just like Cconin(): long keycode; keycode = Crawcin(); long keycode; keycode = Cnecin();

System Functions In addition to character device functions, GEMDOS also contains a number of system control functions. These include routines to manage system memory, to execute and terminate programs, and to get and set the DOS clock and calendar. The first of these allow you to change the 68000 processor’s privilege mode, or to find what the current mode is. The 68000 allows certain operations to be performed in supervisor mode that cannot be performed from the normal user mode. On the ST, these include reading or writing to system variables stored in memory locations below 2048 ($800), and reading or writing to hardware registers located above 167444482 ($FF80000). The macro name of the function used to switch modes is Super(), and its syntax is: long stack, oldstack; oldstack = Super(stack); The meaning of stack will vary depending on what you want the function to do.

If you set stack to – 1L (OxFFFFFFFF), the function returns a code in oldstack which specifies the current privilege mode. If the value returned in oldstack is 0, the processor is user mode. If the value returned is 1, it’s in supervisor mode. If stack is set to any value other than – 1L, the function will toggle the current privilege mode. If the processor is in user mode, it will be set to supervisor mode, with the supervisor stack set to the address passed in stack. If you wish to make the supervisor stack address the same as that of the user stack, pass a value of 0L in stack. The address of the previous supervisor stack is returned in oldstack. This value should be saved, so you may restore the old supervisor stack value when you switch back to user mode. If the function is called when the processor is in supervisor mode, GEMDOS sets it to user mode and sets the supervisor stack value back to the address passed in stack. This address should be the same one returned in oldstack when you first sent the processor into supervisor mode. It’s important that the supervisor stack exist in memory outside the control of your program, so when your program terminates, the system still has a workable stack area. If you fail to set the supervisor stack back to its original value before your program terminates, you may crash the system.

 An example of proper use of the Super() function may be found in the sample program TOGGLE. S, below.

Memory Management Functions

The management of free memory is another common system function made easier by the DOS commands. When a GEM or TOS application program is first loaded and run, it takes control of the entire application RAM space. As you will see below, however, it may want to give back some of that memory so that other applications, desk accessories, or resident programs may use it. Because an application can never know how much free memory will be available, or where that free memory is, the proper way to gain access to additional memory space is through the system memory manager. When a program wants a hunk of free memory in which to store data the user has input, or that it has retrieved from a disk file, it requests the memory manager to allocate a certain number of bytes. If that amount of memory is available, the memory manager removes it from the free list and passes its starting address to the application. When the application is finished with the memory, it tells the memory manager, which returns it to the free pool. The GEMDOS function used to allocate free memory is known by the macro name Malloc().

This function is called as follows: long address, bytes; address = Malloc(bytes); where bytes specifies the number of bytes of memory to allocate. If bytes is set to – 1L (OxFFFFFFFF), the function simply returns the size of the largest block of free memory in address. Otherwise, the function tries to allocate the amount of memory specified. If it is able to allocate the requested memory block, it returns the starting address of the block in address. If there isn’t sufficient free memory to allocate the block that was requested, a value of 0 is returned instead. When the program is finished using the memory it has allocated, it should return it to the system by using the Mfree() call int status; long address; status = Mfree(address);

here address is the starting address of a block of memory allocated with Malloc(). If the memory is successfully returned to the system, a 0 is returned in status. A negative value in status indicates that an error occurred. If a program fails to release one or more blocks of memory, GEMDOS will automatically return them to the free pool when the program terminates. Atari documentation states that if more than 20 blocks of memory are allocated by Malloc() at one time, the memory management system will fail. Since some library functions like fopen() use Malloc() to allocate blocks for their own use, your program should use Malloc() very sparingly. If your program needs more than one memory area allocated, it is best to get one big block of memory and divide it up yourself, rather than requesting many small areas from Malloc(). As stated above, when a program is loaded and run, it takes control of the entire Transient Program Area (TPA), which consists of all of the memory from the program’s Base Page Address on up to the top of free RAM. This area includes the base page, which stores information about the process, the program code, the program data, and program variable storage areas.

Chapter 6

INTRODUCTION The GEM VDI provides a device-independent environment in which you can write graphics applications. This section describes GEM VDI and its architecture. Subsequent sections describe writing an application and all the GEM VDI functions. FEATURES The following features of GEM VDI make it possible for you to write graphics applications that run under several microcomputer operating systems:

• GEM VDI provides a common graphics programming interface that is compatible with the most widely used operating systems, thus making it easy to port many programs
• .GEM VDI provides a device-independent software interface for your application programs. You do not need to rewrite applications for use with different output devices such as screens, printers, and plotters. GEM VDI handles device differences and makes it possible for you to send information to the devices through the application program as if the devices were the same. GEM VDI handles graphics requests and supplies the right driver to run the specific device.
• ENHANCEMENTS GEM VDI includes enhancements’ to GSX functions and now includes the following capabilities: o raster functions–functions that affect raster areas, which are rectangular blocks of pixels on physical devices or rectangular blocks of bits in memory o faces–letter styles stored in dynamically loadable files

6.0

ARCHITECTURE GEM VDI

provides graphics primitives for implementing graphics applications with reduced programming effort. Application programs interface to GEM VDI through a standard calling sequence. Drivers for specific graphics devices translate the standard GEM VDI calls to the unique characteristics of each device. In this way, GEM VDI provides device independence.

GEM VDI is composed of two components: o Graphics Device Operating System (GDOS) o device drivers and face files The GDOS contains the device-independent graphics functions, while the device drivers and face files contain the device-dependent code. GEM VDI is designed in this way to make the principal parts of the GDOS transportable to different hardware configurations. This design also allows applications to run independently of the specific devices con- nected to the system. Graphics Device Operating System (GDOS) The Graphics Device Operating System (GDOS) contains the basic host and device independent graphics functions that can be called by your application program. GDOS provides a standard graphics interface that is constant regardless of specific devices or host hardware, just as the disk operating system standardizes disk interfaces. Your application program accesses the GDOS in much the same way that it accesses’ the operating system.

 The GDOS performs coordinate scaling so that your application can specify points in a normalized space. It uses device-specific information to transform (map) the coordinates into the corresponding values for a particular graphics device. An application can also specify points in raster coordinate space, in which case no transformation occurs

The graphics device drivers are similar to any I/O system. They contain the devicespecific code required to interface your particular graphics devices to the GDOS. The Gevice drivers communicate directly with the graphics devices. GEM VDI requires a unique device driver for each graphics device ina system. A single program can use several graphics devices; the GDOS loads only the appropriate device driver file into memory. By referring to devices with a device identification number, an application program can send graphics information to any one of several memoryresident device drivers. The device driver outputs’ the GEM VDI graphics primitives according to the inherent capabilities of a particular graphics device. In some cases, a device driver emulates standard capabilities not provided by the graphics device hardware. For example, some devices require that dashed lines be simulated by a series of short vectors generated in the device driver.

The GEM VDI package contains drivers for many of the most popular microcomputer-related graphics devices. DEVICE TYPES You can write a GEM VDI-based graphics application for a variety of devices including screens, plotters, printers, and special cameras.

Metafiles A metafile is the stored generic form of a picture file. Any application can create a GEM VDI metafile that can later be called into another graphics application. The metafile driver stores a description of a picture in a data file. These files can later be sent to any device or used to exchange a picture between two applications.

When GEM VDI creates a metafile, it provides the ideal device. Raster Coordinate (RC) and Normalized Device Coordinate (NDC) space are the same (0 to 32767).

 No transform is applied. Refer to “Transforming Points” later in this section for more information on the coordinate spaces. Refer to Appendix C for information about the file format for metafiles. Multiple Workstations The application program specifies the graphics function to be performed by a device driver with an operation code (opcode) in the control array. ““Opcodes” in Section 2 describes the opcodes. Because multiple workstations can be open at the same time, each GEM VDI function must be provided with a unique reference to the Gesired device. This identification is referred to as the device handle.

Device Handles The GDOS assigns the device handle when the Open Workstation function is called by the application program. The Open Workstation call returns the device handle in the array element contr1(6).

All subsequent GEM VDI calls need to supply the device handle as an input in element contrl1(6).

 ASSIGN.SYS The ASSIGN.SYS file is a text file, and can be created or edited using any text editor. The file lists the device driver filenames and face filenames, their device numbers, and device-specific information. The device numbers are assigned according to their type. Refer to Table 1-1 for device numbers.

Table 1-1.

Device Identification

Numbers Device

Type Device Number

Screen 1-10 Plotter

11-20 Printer

21-30 Metafile

31-40 Camera

41-50 Tablet

91-60

APPLICATION PROGRAMS With appropriate calls to the GDOS, you can write application programs in assembly language or ina high-level language that supports the GEM VDI calling conventions. You can compile or assemble and link programs containing GEM VDI calls in the normal manner.

 VIRTUAL DEVICE INTERFACE This guide contains the specification of the GEM Virtual Device Interface (VDI) and defines how applications interface to GEM VDI. The GEM VDI specifies the calling sequence to access device driver functions as well as the necessary calling parameters. Refer to Appendix E for the main entry into the VDI for your operating system. The main entry point into the VDI is a single subroutine with five arguments, in the form of five arrays: control array array of input parameters array of input point coordinates array of output parameters 00000 array of output point coordinates

All computer graphics are displayed using a coordinate system. GEM VDI makes’ sure the coordinate system of one device matches the coordinate system of another. For example, with GEM VDI, the application program produces the same graphics image on a printer aS on ae screen. The linetypes and fill styles are the same in Normalized Device Coordinates (NDC), which are described below. Character sizes are different. The same number of characters are displayed per line, but a printer’s line length is’ generally greater than a screen’s. Transformation Mode The application program can address the display surface using one of two coordinate systems: Oo Normalized Device Coordinates (NDC) o Raster Coordinates (RC) The transformation mode, specified at Open Workstation, determines which coordinate sys- tem is used. Normalized Device Coordinates Normalized Device Coordinates (NDC) address the graphics display independent of the device coordinate size. These units are then mapped to Raster Coordinates by the GDOS. The transformation mode set at Open Workstation determines whether the GDOS maps from NDC units to the Raster Coordinates.

The post Write up on Tech geek history: Atari 1979 Operating System (2nd Publication Revision) Chapter 1 to Chapter 6 appeared first on DayDreamin’ Comics.

Literature Review

0.1 DBI – The Database Interface for Perl 5

DBI, the Database Interface for Perl 5, is an ongoing effort to provide a robust database-independent interface for database connectivity that abstracts the ‘guts’ of database technologies away from the programmer. With the explosion in popularity of Perl as the de rigeur language for CGI programming, a simple, and standard, connection interface to databases is imperative.

 Chapter 1 : The Architecture of DBI

The architecture of DBI is an elegant one. We are channeled  toward the solution by the very concept of the interface that we are trying to define. Furthermore, Perl, as usual, helps us along by providing powerful syntactic constructs and regular expressions that facilitate data processing on the scale required by large database applications.

The DBI Architecture

The DBI interface is the term used to describe both the interface specification – the methods used to build your programs – and the software modules that make this possible. We’ll first take a look at why you should be using DBI, and then describe the modular structure of DBI and its interaction with Perl. Finally, we’ll show you some sample DBI code.

Why DBI?

Database Independence

Database programming, as you will learn after you’ve programmed or administered a few different ones, is pretty much-of-a-muchness. Hence DBI. The fundamental processes involved in doing anything with a database are pretty similar right across the board: connecting to and disconnecting from the database, opening and closing cursors,( A cursor is a construct identifying a particular set of rows in a database.) and storing and retrieving data. The typical order in which this happens is:

1. Load DBI
2. Connect to database, loading the required driver (DBD)
3. Open a cursor containing a SQL statement.( Pronounced “sequel.” SQL is an ANSI-compliant Standard Query Language for querying and manipulating data within databases.)
4. Fetch the results
5. Close the cursor
6. Disconnect from the database
7. Exit

Now, there are differences between each database engine, like datatypes of fields retrieved, precision, extra non-ANSI-compliant features and so on, but their basic operations are the same from system to system. DBI provides a unified layer so that people can write portable code while still allowing access to the non-standard features as well.

Once we’ve fetched some data from the database, what next? In general, DBI simply provides your data as scalar variables, which you can then manipulate as you would any other scalar.

Another feature is the ability to connect to more than one database simultaneously from within the same Perl program – even (dare we say it?) to databases from different vendors. You can connect to an Oracle and an mSQL and an Informix database at the same time. “What’s the point in that?” I hear you cry. Well, say you have a corporate Oracle database, and you want to display some of its data on the Web. Oracle via CGI can be slow due to Oracle’s internal login procedures, so you’ve decided to use mSQL as well.

Option number 1 is to write a program that runs SQL scripts and dumps the data to a flat file, perhaps comma-separated, which another program then reads and loads into the mSQL database. This is far too much like hard work.

Option number 2 is to write a single Perl script that connects simultaneously to both the Oracle and the mSQL databases, reads the data from Oracle, and puts it into mSQL transparently. Change Oracle to Informix and you won’t need to alter a thing.

Think about that the next time you’re writing two separate programs in C (a well-known excellent string and regular-expression-handling language) with different vendor precompilers on separate platforms. Then think on the paradise of cross-platform portability and cross-database connectivity. You know it makes sense.

DBI is a living organism and, coupled with Perl’s current popularity for CGI scripting language and rapid development, will become a more important factor in decisions to use Perl as a “serious” programming language. Similarities have been drawn between DBI and ODBC( The Open Database Connectivity API. This defines an API for accessing databases based on X/Open and ISO standards. ODBC has taken several years to spread from being Microsoft specific to being shipped with Unix operating systems, such as Solaris 2.5. ODBC is more complex than DBI! ) that have led to questions of “Why do we need DBI?” DBI is simpler than ODBC. DBI will run immediately on more platforms than ODBC, except, rather perversely, the platform ODBC originally sprang from: Microsoft Windows. DBI is free. If this isn’t enough to convince you, DBI will be extended in the future to allow you to use off-the-shelf ODBC drivers anyway.

Databases vs. dbm

Databases, be they Object Database Management Systems (ODBMS) or the more common Relational Database Management Systems (RDBMS) are engineered to store and retrieve data. That’s their prime purpose. Unix was originally blessed with simple file-based “databases”, namely the dbm system. dbm lets you store data in files, and retrieve that data quickly. However, it also has two notable drawbacks:

File Locking. The dbm systems did not allow particularly robust file locking capabilities, nor any capability for correcting problems arising from simultaneous writes.

Arbitrary Data Structures. The dbm systems only allows a single fixed data structure: key-value pairs. That value could be a complex object, such as a struct, but the key had to be unique. This was a large limitation on the usefulness of dbm systems.

However, the dbm systems still provide a useful function for users with simple datasets and limited resources, since they’re fast, robust, and extremely well-tested. Perl modules to access dbm systems have now been integrated into the core Perl distribution via the AnyDBM_File module.

The Modules

Perl 5 has a powerful mechanism to ‘plug’ external modules into the Perl interpreter. This mechanism is realized by actually compiling and linking the module into the Perl interpreter, or by dynaloading (dynamically loading) the module into a running interpreter only when needed. This notion of separable modules is central to the philosophy of both Perl and DBI.

DBI essentially acts as a conduit for the DBD (Database Driver) modules. The DBDs all implement the methods defined in DBI, (e.g. connect()), but in a database-specific way. To clarify this somewhat: Since you, the DBI user, wish to use a completely database-independent programming layer, some part of the system must know how to execute the database-dependent code. That’s what the DBD does. The application programmer need never know the DBD is there! All they will be aware of is the database-independent methods defined by DBI. The DBD code is written and maintained by many volunteers and now cover a fairly broad base of database vendors, including Oracle, Informix, mSQL, Ingres, and Sybase. The Web pages listed at the end of this article contain more detailed and up-to-date information about the state of the different DBDs and the DBI as a whole.

Before we even start looking at how you use DBI, you need to download and install the modules. You will always need the DBI module itself, as well as one of the DBD modules for whichever database you have installed. You can download DBI and the DBD modules from the CPAN, details of which can be found on page 20. Please follow the instructions in the files prefixed with README. They are important. They are called README for a good reason. Your ability to get civil answers to your questions depends on familiarity with them!

Next, you need to inform the Perl interpreter that, hey, now would be a good time to load the DBI module in. This always happens before any other DBI work, such as loading one of the database vendor-specific libraries, or attempting to connect to a database should happen. Now, this seems quite obvious, but if I had a dollar for each time….

#!/usr/bin/perl -w

use DBI;

And that’s it! Honest. If you don’t believe me, try this (assuming you’re running a Bourne-compatible shell):

$ PERL_DL_DEBUG=2 perl -e ‘use DBI;’

DynaLoader.pm loaded (/usr/local/lib/perl5

/i486-linux/5.003/usr/local/lib/perl5/usr/

local/lib/perl5/site_perl/i486-linux/usr/

local/lib/perl5/site_perl . /usr/local/lib

/usr/local/lib /lib /usr/lib)

DynaLoader::bootstrap for DBI (auto/DBI/DBI.so)

See. That simple statement dynamically loads the shared library containing the DBI code into the interpreter, and imports the DBI interface methods, which means we can now start using DBI in anger.

If you refer back to the Architecture diagram on page 15, you can clearly see that all access to the databases are marshalled, or funnelled, through the DBI module. Therefore, we need to make Perl aware of our DBDs. and to do this, we need to use DBI.

To simplify matters, the DBI hides the details of loading the drivers. As you connect to a database with DBI->connect(), the DBI makes sure an appropriate driver is loaded and passes on the request to it. For example:

#!/usr/bin/perl -w

use DBI;

$dbh = DBI->connect( ‘connection_string’, ‘username’,                    

                        ‘password’,          ‘mSQL’ );

die “Can’t connect to database: $DBI::errstr\n” unless $dbh;

loads the DBI driver and then the mSQL driver (since it’s not already loaded) and then attempts a connection to the specified database. This call returns a database handle, which we’ll see more of later. To use this method for other DBDs, simply change the fourth argument from mSQL to your database engine. If not specified, the value of the DBI_DRIVER environment variable will be used instead.

The connect method will croak() if the driver can’t be installed. Otherwise it’ll return UNDEF on any other error and $DBI::errstr will contain an error message. The available_drivers() method returns a list of DBI drivers currently installed. Specific drivers can be loaded using the install_driver() method.

Handles

Handles are Perl objects returned by various DBI methods which the programmer can use to access data at various abstracted layers. The handles that are used by DBI are as follows and can be seen in the figure below.

Driver Handles. A Driver Handle, or drh, encapsulates the database driver itself. The driver handle does not connect you to a database, nor does it let you perform any database operations. It merely acts as a conduit between the DBI and the low-level database API calls. Generally you won’t need to deal with driver handles. The DBI does that.

Database Handles. A Database Handle, or dbh, encapsulates a single connection to a given database via a driver handle. There can be any number of database handles per driver handle. For example, if we have a script that copies data from one database to another where both databases are mSQL, then we will have one driver handle but two database handles. In our earlier scenario with a mSQL database and an Oracle database, we would have two driver handles, each with a single database handle.

Statement Handles. A Statement Handle, or sth, encapsulates a command issued to a database via a database handle. As with database handles, there can be any number of statement handles per database handle – or at least as many as the vendor permits. For example, if we have two tables in our database, one containing data and the other containing a stale copy of the data, and we have a program that refreshes the stale copy from the original, we could use two statement handles, one to SELECT the data from the first table, the second to UPDATE the data in the other table. These statement handles would operate asynchronously, if desired, and with the advent of multi-threading, this asynchronous behavior will become extremely powerful. Cursors are simply statement handles for SELECT statements.

Emulation Layers

Some of the most commonly asked questions on the comp.lang.perl.* newsgroups and DBI mailing lists are “I can’t get Oraperl to compile for Perl 5. It only seems to work with Perl 4. What do I do?” and “I’ve got all this Oraperl/Ingperl/isqlperl stuff and I want to upgrade to DBI, but I don’t want to recode it all. What can I do?”

Well, as per usual, we’re ahead of you. DBD::Oracle was released originally with an Oraperl emulation layer, a layer of software that translates the original Oraperl API calls into DBI method invocations. The upshot: your existing Oraperl code will now work transparently using DBI and DBD::Oracle, which means you can now start writing new code using the DBI interface, whilst either maintaining the Oraperl code, or migrating it to DBI.

The DBD::mSQL developers are about to release an Msqlperl emulation layer, and there is an isqlperl emulation layer being developed for DBD::Informix.

DBI Past and Future

The volunteer work to formulate a database independent interface for Perl started way back in September 1992, when it was known as DBperl. By the time a Perl 4 version of the specification was just about settled in early 1994, Perl 5 was arriving. Rather than rework the draft specification, an object oriented prototype DBI and DBD::Oracle driver were implemented and released in October 1994 by Tim Bunce. As the modules matured, many volunteers developed drivers for other databases.

The DBI is not without problems. Chief among them is a lack of current documentation. Work is underway to revise the DBperl specification (which, even though rather old, is still a detailed and interesting document). Another issue is the ‘narrow’ interface currently offered by the DBI. The current minimal functionality is actually intentional, at least in part. It will help to ensure that existing DBI code continues to work correctly when ODBC support is added. ODBC support is the next big item on the DBI agenda. The plan is to add ODBC support into the DBI so that ODBC drivers can be loaded and used in the same way, and at the same time, as the existing DBD drivers. The DBI will also adopt more ODBC/ISO standard conventions, such as the numeric values used to describe data types.

Everywhere you find data, you find databases. At the simplest level, every file can be considered a database. At the most complex level, expensive and complex relational database systems handle thousands of transactions per second. In between are countless improvised schemes for fast access to loosely structured data. Perl can work with all of them. Early in the history of computers, people noticed that flat file databases don’t scale to large data sets. Flat files were tamed using fixed-length records or auxiliary indices, but updating became expensive, and previously simple applications bogged down with I/O overhead. After some head-scratching, clever programmers devised a better solution. As hashes in memory provide more flexible access to data than do arrays, hashes on disk offer more convenient kinds of access than array-like text files. These benefits in access time cost you space, but disk space is cheap these days (or so the reasoning goes). The DBM library gives Perl programmers a simple, easy-to-use database. You use the same standard operations on hashes bound to DBM files as you do on hashes in memory. In fact, that’s how you use DBM databases from Perl. You call dbmopen with the name of a hash and the filename holding the database. Then whenever you access the hash, Perl consults or changes the DBM database on disk

Perl Cookbook

Chapter 2 -Files & Records in Perl

A flat file is essentially a plain text file that does not contain any structured relationships with other files or databases. This type of file can be as simple as a text document, a CSV (Comma-Separated Values) file, an Excel spreadsheet, or even a binary file. A flat file database consists of just one table, with the size depending on the amount of data it needs to hold. An everyday example could be a list of names, addresses, and phone numbers—this basic data structure can be considered a flat file database.

The concept of flat file databases dates back to the earliest days of computing, arguably beginning with the punch card files created by Herman Hollerith in the 1890s for the US Census Bureau. These early databases did not feature relational aspects, such as indexing or linking of records. Despite the evolution of data storage and management technologies, flat file formats like CSV have remained popular due to their simplicity, efficiency, and ease of use.

Flat files are particularly valued for their ability to facilitate the import and export of data between different software applications or systems. They are stored in a simple text format where each line represents a distinct record, and fields within a record are separated by a specific delimiter, like a comma or tab. This simplicity ensures wide compatibility with various software applications and makes flat files an accessible and flexible option for data transfer tasks without requiring specialized software or database management systems.

Moreover, flat files are easy to create, edit, and customize using basic text editors, allowing users to adjust the structure and content of these files to meet specific data import or export requirements. The straightforward nature of flat files also contributes to their continued use in modern computing environments, from simple lists utilized in web interactions to complex data transfers between disparate systems.

What is a Flat File used for?

Flat files are often used for importing and exporting data because they are a simple and flexible way to transfer data between different software applications or systems. Flat files are typically stored in plain text format, with each line representing a single record, and each record consisting of a set of fields separated by a delimiter such as a comma or a tab.

This format is widely supported by many software applications and can be easily read and processed by a variety of programming languages and tools. Moreover, flat files do not require any special software or database management systems to work with, making them a convenient and lightweight option for data transfer tasks.

Flat files are also easy to create and edit using a simple text editor, which means that users can quickly generate new files or modify existing ones as needed. This makes it possible to customize the structure and content of flat files to meet specific import or export requirements, such as selecting specific fields or records to include or exclude, or formatting the data in a particular way.

What is a Flat File, or Flat File Database?

2.1 Flat-File Databases

The simplest type of database that we can create and manipulate is the old standby, the flat-file database. This database is essentially a file, or group of files, that contains data in a known and standard format that a program scans for the requested information. Modifications to the data are usually done by updating an in-memory copy of the data held in the file, or files, then writing the entire set of data back out to disk. Flat-file databases are typically ASCII text files containing one record of information per line. The line termination serves as the record delimiter.

In this section we’ll be examining the two main types of flat-file database: files that separate fields with a delimiter character, and files that allocate a fixed length to each field. We’ll discuss the pros and cons of each type of data file and give you some example code for manipulating them.

The most common format used for flat-file databases is probably the delimited file in which each field is separated by a delimiting character. And possibly the most common of these delimited formats is the comma-separated values (CSV) file, in which fields are separated from one another by commas. This format is understood by many common programs, such as Microsoft Access and spreadsheet programs. As such, it is an excellent base-level and portable format useful for sharing data between applications.[8]

[8]More excitingly, a DBI driver called DBD::CSV exists that allows you to write SQL code to manipulate a flat file containing CSV data.

Other popular delimiting characters are the colon ( : ), the tab, and the pipe symbol ( | ). The Unix /etc/passwd file is a good example of a delimited file with each record being separated by a colon. Figure 2-1 shows a single record from an /etc/passwd file.

2.2. Querying Data

Since delimited files are a very low-level form of storage manager, any manipulations that we wish to perform on the data must be done using operating system functions and low-level query logic, such as basic string comparisons. The following program illustrates how we can open a data file containing colon-separated records of megalith data, search for a given site, and return the data if found:

The downside to delimited file formats is that if any piece of data contains the delimiting character, you need to be especially careful not to break up the records in the wrong place. Using the Perl split() function with a simple regular expression, as used above, does not take this into account and could produce wrong results. For example, a record containing the following information would cause the split() to happen in the wrong place:

Stonehenge:Wiltshire:SU 123 400:Stone Circle and Henge:Stonehenge: The most famous stone circle

The easiest quick-fix technique is to translate any delimiter characters in the string into some other character that you’re sure won’t appear in your data. Don’t forget to do the reverse translation when you fetch the records back.

Another common way of storing data within flat files is to use fixed-length records in which to store the data. That is, each piece of data fits into an exactly sized space in the data file. In this form of database, no delimiting character is needed between the fields. There’s also no need to delimit each record, but we’ll continue to use ASCII line termination as a record delimiter in our examples because Perl makes it very easy to work with files line by line.

Using fixed-width fields is similar to the way in which data is organized in more powerful database systems such as an RDBMS. The pre-allocation of space for record data allows the storage manager to make assumptions about the layout of the data on disk and to optimize accordingly. For our megalithic data purposes, we could settle on the data sizes of:[10]

[10]The fact that these data sizes are all powers of two has no significance other than to indicate that the authors are old enough to remember when powers of two were significant and useful sometimes. They generally aren’t anymore.

Field           Required Bytes

—–           ————–

Name            64

Location        64

Map Reference   16

Type            32

Description     256

Storing the data in this format requires slightly different storage manager logic to be used, although the standard Perl file I/O functions are still applicable. To test this data for the correct record, we need to implement a different way of extracting the fields from within each record. For a fixed-length data file, the Perl function unpack() is perfect. The following code shows how the unpack() function replaces the split() used above:

### Break up the record data into separate fields

### using the data sizes listed above

( $name, $location, $mapref, $type, $description ) =

    unpack( “A64 A64 A16 A32 A256”, $_ );

Although fixed-length fields are always the same length, the data that is being put into a particular field may not be as long as the field. In this case, the extra space will be filled with a character not normally encountered in the data or one that can be ignored. Usually, this is a space character (ASCII 32) or a nul (ASCII 0).

In the code above, we know that the data is space-packed, and so we remove any trailing space from the name record so as not to confuse the search. This can be simply done by using the uppercase A format with unpack().

If you need to choose between delimited fields and fixed-length fields, here are a few guidelines:

The main limitations

The main limitation with delimited fields is the need to add special handling to ensure that neither the field delimiter or the record delimiter characters get added into a field value.

The main limitation with fixed-length fields is simply the fixed length. You need to check for field values being too long to fit (or just let them be silently truncated). If you need to increase a field width, then you’ll have to write a special utility to rewrite your file in the new format and remember to track down and update every script that manipulates the file directly.

Space

A delimited-field file often uses less space than a fixed-length record file to store the same data, sometimes very much less space. It depends on the number and size of any empty or partially filled fields. For example, some field values, like web URLs, are potentially very long but typically very short. Storing them in a long fixed-length field would waste a lot of space.

While delimited-field files often use less space, they do “waste” space due to all the field delimiter characters. If you’re storing a large number of very small fields then that might tip the balance in favor of fixed-length records.

Speed

These days, computing power is rising faster than hard disk data transfer rates. In other words, it’s often worth using more space-efficient storage even if that means spending more processor time to use it.

Generally, delimited-field files are better for sequential access than fixed-length record files because the reduced size more than makes up for the increase in processing to extract the fields and handle any escaped or translated delimiter characters.

However, fixed-length record files do have a trick up their sleeve: direct access. If you want to fetch record 42,927 of a delimited-field file, you have to read the whole file and count records until you get to the one you want. With a fixed-length record file, you can just multiply 42,927 by the total record width and jump directly to the record using seek().

Furthermore, once it’s located, the record can be updated in-place by overwriting it with new data. Because the new record is the same length as the old, there’s no danger of corrupting the following record.

2.3 Inserting Data

Inserting data into a flat-file database is very straightforward and usually amounts to simply tacking the new data onto the end of the data file. For example, inserting a new megalith record into a colon-delimited file can be expressed as simply as:

#!/usr/bin/perl -w

#

# ch02/insertmegadata/insertmegadata: Inserts a new record into the

#                                     given megalith data file as

#                                     colon-separated data

#

### Check the user has supplied an argument to scan for

###     1) The name of the file containing the data

###     2) The name of the site to insert the data for

###     3) The location of the site

###     4) The map reference of the site

###     5) The type of site

###     6) The description of the site

die “Usage: insertmegadata”

    .” <data file> <site name> <location> <map reference> <type> <description>\n”

    unless @ARGV == 6;

my $megalithFile    = $ARGV[0];

my $siteName        = $ARGV[1];

my $siteLocation    = $ARGV[2];

my $siteMapRef      = $ARGV[3];

my $siteType        = $ARGV[4];

my $siteDescription = $ARGV[5];

### Open the data file for concatenation, and die upon failure

open MEGADATA, “>>$megalithFile”

    or die “Can’t open $megalithFile for appending: $!\n”;

### Create a new record

my $record = join( “:”, $siteName, $siteLocation, $siteMapRef,

                        $siteType, $siteDescription );

### Insert the new record into the file

print MEGADATA “$record\n”

    or die “Error writing to $megalithFile: $!\n”;

### Close the megalith data file

close MEGADATA

    or die “Error closing $megalithFile: $!”;

print “Inserted record for $siteName\n”;

exit;

This example simply opens the data file in append mode and writes the new record to the open file. Simple as this process is, there is a potential drawback. This flat-file database does not detect the insertion of multiple items of data with the same search key. That is, if we wanted to insert a new record about Stonehenge into our megalith database, then the software would happily do so, even though a record for Stonehenge already exists.

This may be a problem from a data integrity point of view. A more sophisticated test prior to appending the data might be worth implementing to ensure that duplicate records do not exist. Combining the insert program with the query program above is a straightforward approach.

Another potential (and more important) drawback is that this system will not safely handle occasions in which more than one user attempts to add new data into the database. Since this subject also affects updating and deleting data from the database, we’ll cover it more thoroughly in a later section of this chapter.

Inserting new records into a fixed-length data file is also simple. Instead of printing each field to the Perl file handle separated by the delimiting character, we can use the pack() function to create a fixed-length record out of the data.

2.4 Updating Data

Updating data within a flat-file database is where things begin to get a little more tricky. When querying records from the database, we simply scanned sequentially through the database until we found the correct record. Similarly, when inserting data, we simply attached the new data without really knowing what was already stored within the database.

The main problem with updating data is that we need to be able to read in data from the data file, temporarily mess about with it, and write the database back out to the file without losing any records.

One approach is to slurp the entire database into memory, make any updates to the in-memory copy, and dump it all back out again. A second approach is to read the database in record by record, make any alterations to each individual record, and write the record immediately back out to a temporary file. Once all the records have been processed, the temporary file can replace the original data file. Both techniques are viable, but we prefer the latter for performance reasons. Slurping entire large databases into memory can be very resource-hungry.

The following short program implements the latter of these strategies to update the map reference in the database of delimited records:

#!/usr/bin/perl -w

#

# ch02/updatemegadata/updatemegadata: Updates the given megalith data file

#                                     for a given site. Uses colon-separated

#                                     data and updates the map reference field.

#

### Check the user has supplied an argument to scan for

###     1) The name of the file containing the data

###     2) The name of the site to search for

###     3) The new map reference

die “Usage: updatemegadata <data file> <site name> <new map reference>\n”

    unless @ARGV == 3;

my $megalithFile = $ARGV[0];

my $siteName     = $ARGV[1];

my $siteMapRef   = $ARGV[2];

my $tempFile     = “tmp.$$”;

### Open the data file for reading, and die upon failure

open MEGADATA, “<$megalithFile”

    or die “Can’t open $megalithFile: $!\n”;

### Open the temporary megalith data file for writing

open TMPMEGADATA, “>$tempFile”

    or die “Can’t open temporary file $tempFile: $!\n”;

### Scan through all the records looking for the desired site

while ( <MEGADATA> ) {

    ### Quick pre-check for maximum performance:

    ### Skip the record if the site name doesn’t appear as a field

    next unless m/^\Q$siteName:/;

    ### Break up the record data into separate fields

    ### (we let $description carry the newline for us)

    my ( $name, $location, $mapref, $type, $description ) =

         split( /:/, $_ );

    ### Skip the record if the site name doesn’t match. (Redundant after the

    ### reliable pre-check above but kept for consistency with other examples.)

    next unless $siteName eq $name;

    ### We’ve found the record to update, so update the map ref value

    $mapref = $siteMapRef;

    ### Construct an updated record

    $_ = join( “:”, $name, $location, $mapref, $type, $description );

}

continue {

   ### Write the record out to the temporary file

   print TMPMEGADATA $_

      or die “Error writing $tempFile: $!\n”;

}

### Close the megalith input data file

close MEGADATA;

### Close the temporary megalith output data file

close TMPMEGADATA

   or die “Error closing $tempFile: $!\n”;

### We now “commit” the changes by deleting the old file…

unlink $megalithFile

   or die “Can’t delete old $megalithFile: $!\n”;

### and renaming the new file to replace the old one.

rename $tempFile, $megalithFile

   or die “Can’t rename ‘$tempFile’ to ‘$megalithFile’: $!\n”;

exit 0;

You can see we’ve flexed our Perl muscles on this example, using a while … continue loop to simplify the logic and adding a pretest for increased speed.

An equivalent program that can be applied to a fixed-length file is very similar, except that we use a faster in-place update to change the contents of the field. This principle is similar to the in-place query described previously: we don’t need to unpack and repack all the fields stored within each record, but can simply update the appropriate chunk of each record. For example:

### Scan through all the records looking for the desired site

while ( <MEGADATA> ) {

    ### Quick pre-check for maximum performance:

    ### Skip the record if the site name doesn’t appear at the start

    next unless m/^\Q$siteName/;

    ### Skip the record if the extracted site name field doesn’t match

    next unless unpack( “A64”, $_ ) eq $siteName;

    ### Perform in-place substitution to upate map reference field

    substr( $_, 64+64, 16) =  pack( “A16”, $siteMapRef ) );

}

This technique is faster than packing and unpacking each record stored within the file, since it carries out the minimum amount of work needed to change the appropriate field values.

You may notice that the pretest in this example isn’t 100% reliable, but it doesn’t have to be. It just needs to catch most of the cases that won’t match in order to pay its way by reducing the number of times the more expensive unpack and field test gets executed. Okay, this might not be a very convincing application of the idea, but we’ll revisit it more seriously later in this chapter.

2.5 Deleting Data

The final form of data manipulation that you can apply to flat-file databases is the removal, or deletion, of records from the database. We shall process the file a record at a time by passing the data through a temporary file, just as we did for updating, rather than slurping all the data into memory and dumping it at the end.

With this technique, the action of removing a record from the database is more an act of omission than any actual deletion. Each record is read in from the file, tested, and written out to the file. When the record to be deleted is encountered, it is simply not written to the temporary file. This effectively removes all trace of it from the database, albeit in a rather unsophisticated way.

The following program can be used to remove the relevant record from the delimited megalithic database when given an argument of the name of the site to delete:

#!/usr/bin/perl -w

#

# ch02/deletemegadata/deletemegadata: Deletes the record for the given

#                                     megalithic site. Uses

#                                     colon-separated data

#

### Check the user has supplied an argument to scan for

###     1) The name of the file containing the data

###     2) The name of the site to delete

die “Usage: deletemegadata <data file> <site name>\n”

    unless @ARGV == 2;

my $megalithFile  = $ARGV[0];

my $siteName      = $ARGV[1];

my $tempFile      = “tmp.$$”;

### Open the data file for reading, and die upon failure

open MEGADATA, “<$megalithFile”

    or die “Can’t open $megalithFile: $!\n”;

### Open the temporary megalith data file for writing

open TMPMEGADATA, “>$tempFile”

    or die “Can’t open temporary file $tempFile: $!\n”;

### Scan through all the entries for the desired site

while ( <MEGADATA> ) {

    ### Extract the site name (the first field) from the record

    my ( $name ) = split( /:/, $_ );

    ### Test the sitename against the record’s name

    if ( $siteName eq $name ) {

        ### We’ve found the record to delete, so skip it and move to next record

        next;

    }

    ### Write the original record out to the temporary file

    print TMPMEGADATA $_

        or die “Error writing $tempFile: $!\n”;

    }

### Close the megalith input data file

close MEGADATA;

### Close the temporary megalith output data file

close TMPMEGADATA

    or die “Error closing $tempFile: $!\n”;

### We now “commit” the changes by deleting the old file …

unlink $megalithFile

    or die “Can’t delete old $megalithFile: $!\n”;

### and renaming the new file to replace the old one.

rename $tempFile, $megalithFile

    or die “Can’t rename ‘$tempFile’ to ‘$megalithFile’: $!\n”;

exit 0;

The code to remove records from a fixed-length data file is almost identical. The only change is in the code to extract the field value, as you’d expect:

### Extract the site name (the first field) from the record

my ( $name ) = unpack( “A64”, $_ );

2.6 Summary of Flat-File Databases

The main benefit of using flat-file databases for data storage is that they can be fast to implement and fast to use on small and straightforward datasets, such as our megalithic database or a Unix password file.

The code to query, insert, delete, and update information in the database is also extremely simple, with the parsing code potentially shared among the operations. You have total control over the data file formats, so that there are no situations outside your control in which the file format or access API changes. The files are also easy to read in standard text editors (although in the case of the Storable example, they won’t make very interesting reading).

The downsides of these databases are quite apparent. As we’ve mentioned already, the lack of concurrent access limits the power of such systems in a multi-user environment. They also suffer from scalability problems due to the sequential nature of the search mechanism. These limitations can be coded around (the concurrent access problem especially so), but there comes a point where you should seriously consider the use of a higher-level storage manager such as DBM files. DBM files also give you access to indexed data and allow nonsequential querying.

Before we discuss DBM files in detail, the following sections give you examples of more sophisticated management tools and techniques, as well as a method of handling concurrent users.

Putting Complex Data into Flat Files (Programming the Perl DBI)

2.7 Record Types in Perl

Reading simple record-oriented data Perl makes it very easy to deal with record-oriented data, particularly simple records of the type we are discussing here. We have seen before the idiom where you can read a file a line at a time using a construct like

while ()

{ chomp; # remove newline # each line in turn is assigned to $_ }

The most important part of the construct is the use of to read data from the file handle FILE which has presumably been assigned to a file earlier in the program by a call to the open function. This file input operator can return two different results, depending on whether it is used in scalar context or array context. When called in a scalar context, the file input operator returns the next record from the file handle. This begs the obvious question of what constitutes a record. The answer is that input records are separated by a sequence of characters called (logically enough) the input record separator. This value is stored in the variable $/.

The default value is a newline \n (which is translated to the appropriate actual characters for the specific operating system), but this can be altered to any other string of characters. We will look at this usage in more detail later, but for now the default value will suffice. When called in an array context, the file input operator returns a list in which each element is a record from the input file. You can, therefore, call the file input operator in one of these two ways:

my $next_line = ;

my @whole_file = ;

In both of these examples it is important to realize that each record—whether it is the record stored in $next_line

or one of the records in

 @whole_file—will still contain the value of $/ at the end.1

Often you will want to get rid of this and the easiest way to do it is by using the chomp function. chomp is passed either a scalar or an array and removes the

value of $/

from the end of the scalar or each element of the array.

 If no argument is passed to chomp then it works on $_. 2 Reading data a record at a time (from first principles) Now that we understand a little more about the file input operator and chomp, let’s see if we can build our standard data munging input construct from first principles. A first attempt at processing each line in a file might look something like this

my $line;

 while ($line = )

 { chomp $line; … }

Fixed-width data is becoming less common, but it is still possible that you will come across it, particularly if you are exchanging data with an older computer system that runs on a mainframe or is written in

2.8 COBOL

Reading fixed-width data In a fixed-width data record, there is nothing to distinguish one data item from the next one. Each data item is simply written immediately after the preceding one, after padding it to a defined width. This padding can either be with zeroes or with spaces and can be before or after the data.1 In order to interpret the data, we need more information about the way it has been written. This is normally sent separately from the data itself but, as we shall see later, it is also possible to encode this data within the files. Here is an example of two fixed-width data records: 00374

Bloggs & Co 19991105100103+00015000 00375

Smith Brothers 19991106001234-00004999

Multiple record types One slight variation of the fixed-width data record has different sets of data fields for different types of data within the same file. Consider a system that maintains a product list and, at the end of each day, produces a file that lists all new products added and old products deleted during the day. For a new product, you will need a lot of data (perhaps product code, description, price, stock count and supplier identifier). For the deleted product you only need the product code (but you might also list the product description to make the report easier to follow).

 Each record will have some kind of identifier and the start of the line denoting which kind of record it is. In our example they will be the strings ADD and DEL. Here are some sample data:

ADD0101Super Widget 00999901000SUPP01

ADD0102Ultra Widget 01499900500SUPP01

DEL0050Basic Widget DEL0051Cheap Widget

On the day covered by this data, we have added two new widgets to our product catalogue. The Super Widget (product code 0101) costs $99.99 and we have 1000 in stock. The Ultra Widget (product code 0102) costs $149.99 and we have 500 in stock. We purchase both new widgets from the same supplier. At the same time we have discontinued two older products, the Basic Widget (Product Code 0050) and the Cheap Widget (Product Code 0051).

Data with no end-of-record marker Another difference that you may come across with fixed-width data is that sometimes it comes without a defined end-of-record marker. As both the size of each field in a record and the number of fields in a record are well defined, we know how long each record will be. It is, therefore, possible to send the data as a stream of bytes and leave it up to the receiving program to split the data into individual records. Perl, of course, has a number of ways to do this. You could read the whole file into memory and split the data using

 substr or unpack,

but for many tasks the amount of data to process makes this unfeasible. The most efficient way is to use a completely different method of reading your data. In addition to the syntax that reads data from file handles one record at a time, Perl supports a more traditional syntax using the read and seek functions. The read function takes three or four arguments. The first three are: a file handle to read data from, a scalar variable to read the data into, and the maximum number of bytes to read. The fourth, optional, argument is an offset into the variable where you want to start writing the data (this is rarely used). read returns the number of bytes read (which can be less than the requested number if you are near the end of a file) and zero when there is no more data to read. Each open file handle has a current position associated with it called a file pointer and read takes its data from the file pointer and moves the pointer to the end of the data it has read. You can also reposition the file pointer explicitly using the seek function. seek takes three arguments: the file handle, the offset you wish to move to, and a value that indicates how the offset should be interpreted.

 If this value is 0 then the offset is calculated from the start of the file, if it is 1 the offset is calculated from the current position, and if it is 2 the offset is calculated from the end of the file. You can always find out the current position of the file pointer by using tell, which returns the offset from the start of the file in bytes. seek and tell are often unnecessary when handling ASCII fixed-width data files, as you usually just read the file in sequential order. Example: reading data with no end-of-record markers using read As an example, if our previous data file were written without newlines, we could use code like this to read it (obviously we could use any of the previously discussed techniques to split the record up once we have read it):

my $template = ‘A5A20A8A6AA8’;

my $data;

while (read STDIN, $data, 48) {

 my @rec = unpack($template, $data);

2.9. ODBC Drivers

Open Database Connectivity, commonly known as ODBC , defines a database-independent API. The ODBC Driver Manager supports the API and manages one or more plug-in drivers for talking to different types of databases. The architecture is shown in Figure

The ODBC architecture Doesn’t this all sound rather familiar? The ODBC driver manager and drivers are doing just what the DBI and its drivers are doing: defining and implementing a database-independent, applicationprogramming interface.

 This leads us to a whole bunch of questions: What’s the difference?

Why not just use ODBC and not the DBI? Can the DBI and ODBC work together? What advantages does one have over the other?

Before we try to answer those questions, let’s get some perspective by looking into the history and goals of ODBC and the DBI, and take a look at the Win32::ODBC module, which implements a direct interface to ODBC for Perl

 ODBC – Embraced and Extended In the early 1990s, a consortium of vendors formed the SQL Access Group to support SQL interoperability across disparate systems. In October 1992 and October 1993, a major part of that work was published as a draft standard entitled ” Call Level Interface,” or CLI, which is another name for an Application Programing Interface, or API. However, the SQL Access Group CLI standard never really took off. At least, not in that form. Microsoft needed to implement a similar concept, to avoid having to release multiple versions of any product that needed to talk to multiple databases.

They saw the SQL Access Group CLI standard and ”embraced and extended” it, radically. The result was the Open Database Connectivity interface, which rapidly became a de facto standard. In fairness, Microsoft turned an income

main features of ODBC that set it apart from the DBI and that enable it to work well as a database-independent interface. The four main features are:

• A standard SQL syntax

• Standard error codes

• Rich metadata

 • Many attributes and options

A Standard SQL Syntax Standardized SQL syntax is something of a Holy Grail. ODBC drivers generally do a good job of implementing it, whereas the DBI just ducks the issue entirely! The problem is that while SQL may be a standardized language in theory, in practice it’s far enough from the standard on most vendors’ databases to cause portability problems.

For example, even a simple task like concatenating two database fields needs to be written like this (for databases conforming to the SQL-92 standard):

SELECT first_name || ‘ ‘ || last_name FROM table Other databases require one of these forms: SELECT first_name + ‘ ‘ + last_name FROM table SELECT CONCAT(first_name, ‘ ‘, last_name) FROM table SELECT CONCAT(CONCAT(first_name, ‘ ‘) last_name) FROM table SELECT first_name CONCAT ‘ ‘ CONCAT last_name FROM table

The SQL dialect used by different database systems is riddled with such inconsistencies, not to mention endless “extensions” to the standard. This is a major headache for developers wishing to write an application that will work with any of a number of databases. The ODBC approach to this problem is rather elegant. It allows portability when using standard SQL, but doesn’t prevent access to database-specific features.

ODBC defines a large number of standard error codes that you can use to determine in reasonable detail what went wrong. They’re not often needed, but when they are, they’re very useful. Having said that, this idyllic picture is tarnished by the fact that many of the codes change depending on the version of the ODBC driver being used. For example, while an ODBC 2.x driver returns “S0011” when a CREATE INDEX statement names an index that already exists, an ODBC 3.x driver returns “42S11”. So much for standards!

The DBI leaves you with having to check for different $DBI::err values or

 $DBI::errstr strings, depending on the database driver being used. The DBI does provide a $DBI::state variable and $h- >state() method that drivers can use to provide the standard error codes, but few do at the moment.

Rich Metadata ODBC defines a wide range of metadata functions that provide information both about the structure of the data in the database and the datatypes supported by the database. The following table lists the functions and shows which are supported by the Win32::ODBC and DBI modules. ODBC Function Win32::ODBC DBI Tables TablePrivileges Columns ColumnPrivileges SpecialColumns Statistics PrimaryKeys ForeignKeys Procedures ProcedureColumns GetTypeInfo As you can see, the DBI lags behind the Win32::ODBC module. By the time you read this book, the DBI may have defined interfaces for some of the functions, but how quickly the drivers actually implement the functions is harder to guess. The DBD::ODBC and DBD::Oracle modules will probably lead the way.

When an application passes an SQL statement to the driver, the driver parses it as an SQL-92 statement, and then rewrites it to match the actual syntax of the database being used. If the parse fails because the SQL doesn’t conform to the standard, then the original SQL is passed to the database unaltered.

 That way, database-specific features can be accessed, and the ODBC parsing doesn’t get in the way. The DBI ducked this whole issue because it would require drivers to be far more complicated than they are now. Parsing and rewriting SQL is not a trivial activity; therefore, the DBI does not try to offer SQL-level portability. In practice, that hasn’t been a big problem for people.

Chapter  3  Perl’s DBI (`Database Interface’) module| Data Manipulation

General information about relational databases

Relational databases started to get to be a big deal in the 1970’s, andthey’re still a big deal today, which is a little peculiar, because they’re a 1960’s technology.

A relational database is a bunch of rectangular tables. Each row of a table is a record about one person or thing; the record contains several pieces of information called fields. Here is an example table:

LASTNAME   FIRSTNAME   ID   POSTAL_CODE   AGE  SEX

Gauss      Karl        119  19107         30   M

Smith      Mark        3    T2V 3V4       53   M

Noether    Emmy        118  19107         31   F

Smith      Jeff        28   K2G 5J9       19   M

Hamilton   William     247  10139         2    M

The names of the fields are LASTNAME, FIRSTNAME, ID, POSTAL_CODE, AGE, and SEX. Each line in the table is a record, or sometimes a row or tuple. For example, the first row of the table represents a 30-year-old male whose name is Karl Gauss, who lives at postal code 19107, and whose ID number is 119.

Sometimes this is a very silly way to store information. When the information naturally has a tabular structure it’s fine. When it doesn’t, you have to squeeze it into a table, and some of the techniques for doing that are more successful than others. Nevertheless, tables are simple and are easy to understand, and most of the high-performance database systems you can buy today operate under this 1960’s model.

About SQL

SQL stands for Structured Query Language. It was invented at IBM in the 1970’s. It’s a language for describing searches and modifications to a relational database.

SQL was a huge success, probably because it’s incredibly simple and anyone can pick it up in ten minutes. As a result, all the important database systems support it in some fashion or another. This includes the big players, like Oracle and Sybase, high-quality free or inexpensive database systems like MySQL, and funny hacks like Perl’s DBD::CSV module, which we’ll see later.

There are four important things one can do with a table:

SELECT
Find all the records that have a certain property

INSERT
Add new records

DELETE
Remove old records

UPDATE
Modify records that are already there

Those are the four most important SQL commands, also called queries. Suppose that the example table above is named people. Here are examples of each of the four important kinds of queries:

SELECT firstname FROM people WHERE lastname = ‘Smith’

(Locate the first names of all the Smiths.)

DELETE FROM people WHERE id = 3

(Delete Mark Smith from the table)

UPDATE people SET age = age+1 WHERE id = 247

(William Hamilton just had a birthday.)

INSERT INTO people VALUES (‘Euler’, ‘Leonhard’, 248, NULL, 58, ‘M’)

(Add Leonhard Euler to the table.)

There are a bunch of other SQL commands for creating and discarding tables, for granting and revoking access permissions, for committing and abandoning transactions, and so forth. But these four are the important ones. Congratulations; you are now a SQL programmer. For the details, go to any reasonable bookstore and pick up a SQL quick reference.

Every database system is a little different. You talk to some databases over the network and make requests of the database engine; other databases you talk to through files or something else.

Typically when you buy a commercial database, you get a library with it. The vendor has written some functions for talking to the database in some language like C, compiled the functions, and the compiled code is the library. You can write a C program that calls the functions in the library when it wants to talk to the database.

Every vendor’s library is different. The names of the functions vary, and the order in which you call them varies, and the details of passing queries to the functions and getting the data back out will vary. Some libraries, like Oracle’s, are very thin&#151they just send the query over to the network to the real database and let the giant expensive real database engine deal with it directly. Other libraries will do more predigestion of the query, and more work afterwards to turn the data into a data structure. Some databases will want you to spin around three times and bark like a chicken; others want you to stand on your head and drink out of your sneaker.

What DBI is For

There’s a saying that any software problem can be solved by adding a layer of indirection. That’s what Perl’s DBI (`Database Interface’) module is all about. It was written by Tim Bunce.

DBI is designed to protect you from the details of the vendor libraries. It has a very simple interface for saying what SQL queries you want to make, and for getting the results back. DBI doesn’t know how to talk to any particular database, but it does know how to locate and load in DBD (`Database Driver’) modules. The DBD modules have the vendor libraries in them and know how to talk to the real databases; there is one DBD module for every different database.

When you ask DBI to make a query for you, it sends the query to the appropriate DBD module, which spins around three times or drinks out of its sneaker or whatever is necessary to communicate with the real database. When it gets the results back, it passes them to DBI. Then DBI gives you the results. Since your program only has to deal with DBI, and not with the real database, you don’t have to worry about barking like a chicken.

Here’s your program talking to the DBI library. You are using two databases at once. One is an Oracle database server on some other machine, and another is a DBD::CSV database that stores the data in a bunch of plain text files on the local disk.

Your program sends a query to DBI, which forwards it to the appropriate DBD module; let’s say it’s DBD::Oracle. DBD::Oracle knows how to translate what it gets from DBI into the format demanded by the Oracle library, which is built into it. The library forwards the request across the network, gets the results back, and returns them to DBD::Oracle. DBD::Oracle returns the results to DBI as a Perl data structure. Finally, your program can get the results from DBI.

On the other hand, suppose that your program was querying the text files. It would prepare the same sort of query in exactly the same way, and send it to DBI in exactly the same way. DBI would see that you were trying to talk to the DBD::CSV database and forward the request to the DBD::CSV module. The DBD::CSV module has Perl functions in it that tell it how to parse SQL and how to hunt around in the text files to find the information you asked for. It then returns the results to DBI as a Perl data structure. Finally, your program gets the results from DBI in exactly the same way that it would have if you were talking to Oracle instead.

There are two big wins that result from this organization. First, you don’t have to worry about the details of hunting around in text files or talking on the network to the Oracle server or dealing with Oracle’s library. You just have to know how to talk to DBI.

Second, if you build your program to use Oracle, and then the following week upper management signs a new Strategic Partnership with Sybase, it’s easy to convert your code to use Sybase instead of Oracle. You change exactly one line in your program, the line that tells DBI to talk to DBD::Oracle, and have it use DBD::Sybase instead. Or you might build your program to talk to a cheap, crappy database like MS Access, and then next year when the application is doing well and getting more use than you expected, you can upgrade to a better database next year without changing any of your code.

There are DBD modules for talking to every important kind of SQL database. DBD::Oracle will talk to Oracle, and DBD::Sybase will talk to Sybase. DBD::ODBC will talk to any ODBC database including Microsoft Acesss. (ODBC is a Microsoft invention that is analogous to DBI itself. There is no DBD module for talking to Access directly.) DBD::CSV allows SQL queries on plain text files. DBD::mysql talks to the excellent MySQL database from TCX DataKonsultAB in Sweden. (MySQL is a tremendous bargain: It’s $200 for commercial use, and free for noncommerical use.)

Example of How to Use DBI

Here’s a typical program. When you run it, it waits for you to type a last name. Then it searches the database for people with that last name and prints out the full name and ID number for each person it finds. For example:

Enter name> Noether

118: Emmy Noether

Enter name> Smith

3: Mark Smith

28: Jeff Smith

Enter name> Snonkopus

No names matched `Snonkopus’.

Enter name> ^D

Here is the code:

use DBI;

my $dbh = DBI->connect(‘DBI:Oracle:payroll’)

or die “Couldn’t connect to database: ” . DBI->errstr;

my $sth = $dbh->prepare(‘SELECT * FROM people WHERE lastname = ?’)

or die “Couldn’t prepare statement: ” . $dbh->errstr;

print “Enter name> “;

while ($lastname = <>) {               # Read input from the user

my @data;

chomp $lastname;

$sth->execute($lastname)             # Execute the query

or die “Couldn’t execute statement: ” . $sth->errstr;

# Read the matching records and print them out

while (@data = $sth->fetchrow_array()) {

my $firstname = $data[1];

my $id = $data[2];

print “\t$id: $firstname $lastname\n”;

}

if ($sth->rows == 0) {

print “No names matched `$lastname’.\n\n”;

}

$sth->finish;

print “\n”;

print “Enter name> “;

}

$dbh->disconnect;

use DBI;

This loads in the DBI module. Notice that we don’t have to load in any DBD module. DBI will do that for us when it needs to.

my $dbh = DBI->connect(‘DBI:Oracle:payroll’);

or die “Couldn’t connect to database: ” . DBI->errstr;

The connect call tries to connect to a database. The first argument, DBI:Oracle:payroll, tells DBI what kind of database it is connecting to. The Oracle part tells it to load DBD::Oracle and to use that to communicate with the database. If we had to switch to Sybase next week, this is the one line of the program that we would change. We would have to change Oracle to Sybase.

payroll is the name of the database we will be searching. If we were going to supply a username and password to the database, we would do it in the connect call:

my $dbh = DBI->connect(‘DBI:Oracle:payroll’, ‘username’, ‘password’)

or die “Couldn’t connect to database: ” . DBI->errstr;

If DBI connects to the database, it returns a database handle object, which we store into $dbh. This object represents the database connection. We can be connected to many databases at once and have many such database connection objects.

If DBI can’t connect, it returns an undefined value. In this case, we use die to abort the program with an error message. DBI->errstr returns the reason why we couldn’t connect&#151“Bad password’’ for example.

my $sth = $dbh->prepare(‘SELECT * FROM people WHERE lastname = ?’)

or die “Couldn’t prepare statement: ” . $dbh->errstr;

The prepare call prepares a query to be executed by the database. The argument is any SQL at all. On high-end databases, prepare will send the SQL to the database server, which will compile it. If prepare is successful, it returns a statement handle object which represents the statement; otherwise it returns an undefined value and we abort the program. $dbh->errstr will return the reason for failure, which might be “Syntax error in SQL’’. It gets this reason from the actual database, if possible.

The ? in the SQL will be filled in later. Most databases can handle this. For some databases that don’t understand the ?, the DBD module will emulate it for you and will pretend that the database understands how to fill values in later, even though it doesn’t.

print “Enter name> “;

Here we just print a prompt for the user.

while ($lastname = <>) {               # Read input from the user

…

}

This loop will repeat over and over again as long as the user enters a last name. If they type a blank line, it will exit. The Perl <> symbol means to read from the terminal or from files named on the command line if there were any.

my @data;

This declares a variable to hold the data that we will get back from the database.

chomp $lastname;

This trims the newline character off the end of the user’s input.

$sth->execute($lastname)             # Execute the query

or die “Couldn’t execute statement: ” . $sth->errstr;

execute executes the statement that we prepared before. The argument $lastname is substituted into the SQL in place of the ? that we saw earlier. execute returns a true value if it succeeds and a false value otherwise, so we abort if for some reason the execution fails.

while (@data = $sth->fetchrow_array()) {

…

}

fetchrow_array returns one of the selected rows from the database. You get back an array whose elements contain the data from the selected row. In this case, the array you get back has six elements. The first element is the person’s last name; the second element is the first name; the third element is the ID, and then the other elements are the postal code, age, and sex.

Each time we call fetchrow_array, we get back a different record from the database. When there are no more matching records, fetchrow_array returns the empty list and the while loop exits.

my $firstname = $data[1];

my $id = $data[2];

These lines extract the first name and the ID number from the record data.

print “\t$id: $firstname $lastname\n”;

This prints out the result.

if ($sth->rows == 0) {

print “No names matched `$lastname’.\n\n”;

}

The rows method returns the number of rows of the database that were selected. If no rows were selected, then there is nobody in the database with the last name that the user is looking for. In that case, we print out a message. We have to do this after the while loop that fetches whatever rows were available, because with some databases you don’t know how many rows there were until after you’ve gotten them all.

$sth->finish;

print “\n”;

print “Enter name> “;

Once we’re done reporting about the result of the query, we print another prompt so that the user can enter another name. finish tells the database that we have finished retrieving all the data for this query and allows it to reinitialize the handle so that we can execute it again for the next query.

$dbh->disconnect;

When the user has finished querying the database, they type a blank line and the main while loop exits. disconnect closes the connection to the database.

Cached Queries

Here’s a function which looks up someone in the example table, given their ID number, and returns their age:

sub age_by_id {

# Arguments: database handle, person ID number

my ($dbh, $id) = @_;

my $sth = $dbh->prepare(‘SELECT age FROM people WHERE id = ?’)

or die “Couldn’t prepare statement: ” . $dbh->errstr;

$sth->execute($id)

or die “Couldn’t execute statement: ” . $sth->errstr;

my ($age) = $sth->fetchrow_array();

return $age;

}

It prepares the query, executes it, and retrieves the result.

There’s a problem here though. Even though the function works correctly, it’s inefficient. Every time it’s called, it prepares a new query. Typically, preparing a query is a relatively expensive operation. For example, the database engine may parse and understand the SQL and translate it into an internal format. Since the query is the same every time, it’s wasteful to throw away this work when the function returns.

Here’s one solution:

{ my $sth;

sub age_by_id {

# Arguments: database handle, person ID number

my ($dbh, $id) = @_;

if (! defined $sth) {

$sth = $dbh->prepare(‘SELECT age FROM people WHERE id = ?’)

or die “Couldn’t prepare statement: ” . $dbh->errstr;

}

$sth->execute($id)

or die “Couldn’t execute statement: ” . $sth->errstr;

my ($age) = $sth->fetchrow_array();

return $age;

}

}

There are two big changes to this function from the previous version. First, the $sth variable has moved outside of the function; this tells Perl that its value should persist even after the function returns. Next time the function is called, $sth will have the same value as before.

Second, the prepare code is in a conditional block. It’s only executed if $sth does not yet have a value. The first time the function is called, the prepare code is executed and the statement handle is stored into $sth. This value persists after the function returns, and the next time the function is called, $sth still contains the statement handle and the prepare code is skipped.

Here’s another solution:

sub age_by_id {

# Arguments: database handle, person ID number

my ($dbh, $id) = @_;

my $sth = $dbh->prepare_cached(‘SELECT age FROM people WHERE id = ?’)

or die “Couldn’t prepare statement: ” . $dbh->errstr;

$sth->execute($id)

or die “Couldn’t execute statement: ” . $sth->errstr;

my ($age) = $sth->fetchrow_array();

return $age;

}

Here the only change to to replace prepare with prepare_cached. The prepare_cached call is just like prepare, except that it looks to see if the query is the same as last time. If so, it gives you the statement handle that it gave you before.

Transactions

Many databases support transactions. This means that you can make a whole bunch of queries which would modify the databases, but none of the changes are actually made. Then at the end you issue the special SQL query COMMIT, and all the changes are made simultaneously. Alternatively, you can issue the query ROLLBACK, in which case all the queries are thrown away.

As an example of this, consider a function to add a new employee to a database. The database has a table called employees that looks like this:

FIRSTNAME  LASTNAME   DEPARTMENT_ID

Gauss      Karl       17

Smith      Mark       19

Noether    Emmy       17

Smith      Jeff       666

Hamilton   William    17

and a table called departments that looks like this:

ID   NAME               NUM_MEMBERS

17   Mathematics        3

666  Legal              1

19   Grounds Crew       1

The mathematics department is department #17 and has three members: Karl Gauss, Emmy Noether, and William Hamilton.

Here’s our first cut at a function to insert a new employee. It will return true or false depending on whether or not it was successful:

sub new_employee {

# Arguments: database handle; first and last names of new employee;

# department ID number for new employee’s work assignment

my ($dbh, $first, $last, $department) = @_;

my ($insert_handle, $update_handle);

my $insert_handle =

$dbh->prepare_cached(‘INSERT INTO employees VALUES (?,?,?)’);

my $update_handle =

$dbh->prepare_cached(‘UPDATE departments

SET num_members = num_members + 1

WHERE id = ?’);

die “Couldn’t prepare queries; aborting”

unless defined $insert_handle && defined $update_handle;

$insert_handle->execute($first, $last, $department) or return 0;

$update_handle->execute($department) or return 0;

return 1;   # Success

}

We create two handles, one for an insert query that will insert the new employee’s name and department number into the employees table, and an update query that will increment the number of members in the new employee’s department in the department table. Then we execute the two queries with the appropriate arguments.

There’s a big problem here: Suppose, for some reason, the second query fails. Our function returns a failure code, but it’s too late, it has already added the employee to the employees table, and that means that the count in the departments table is wrong. The database now has corrupted data in it.

The solution is to make both updates part of the same transaction. Most databases will do this automatically, but without an explicit instruction about whether or not to commit the changes, some databases will commit the changes when we disconnect from the database, and others will roll them back. We should specify the behavior explicitly.

Typically, no changes will actually be made to the database until we issue a commit. The version of our program with commit looks like this:

sub new_employee {

# Arguments: database handle; first and last names of new employee;

# department ID number for new employee’s work assignment

my ($dbh, $first, $last, $department) = @_;

my ($insert_handle, $update_handle);

my $insert_handle =

$dbh->prepare_cached(‘INSERT INTO employees VALUES (?,?,?)’);

my $update_handle =

$dbh->prepare_cached(‘UPDATE departments

SET num_members = num_members + 1

WHERE id = ?’);

die “Couldn’t prepare queries; aborting”

unless defined $insert_handle && defined $update_handle;

my $success = 1;

$success &&= $insert_handle->execute($first, $last, $department);

$success &&= $update_handle->execute($department);

my $result = ($success ? $dbh->commit : $dbh->rollback);

unless ($result) {

die “Couldn’t finish transaction: ” . $dbh->errstr

}

return $success;

}

We perform both queries, and record in $success whether they both succeeded. $success will be true if both queries succeeded, false otherwise. If the queries succeded, we commit the transaction; otherwise, we roll it back, cancelling all our changes.

The problem of concurrent database access is also solved by transactions. Suppose that queries were executed immediately, and that some other program came along and examined the database after our insert but before our update. It would see inconsistent data in the database, even if our update would eventually have succeeded. But with transactions, all the changes happen simultaneously when we do the commit, and the changes are committed automatically, which means that any other program looking at the database either sees all of them or none.

do

If you’re doing an UPDATE, INSERT, or DELETE there is no data that comes back from the database, so there is a short cut. You can say

$dbh->do(‘DELETE FROM people WHERE age > 65’);

for example, and DBI will prepare the statement, execute it, and finish it. do returns a true value if it succeeded, and a false value if it failed. Actually, if it succeeds it returns the number of affected rows. In the example it would return the number of rows that were actually deleted. (DBI plays a magic trick so that the value it turns is true even when it is 0. This is bizarre, because 0 is usually false in Perl. But it’s convenient because you can use it either as a number or as a true-or-false success code, and it works both ways.)

AutoCommit

If your transactions are simple, you can save yourself the trouble of having to issue a lot of commits. When you make the connect call, you can specify an AutoCommit option which will perform an automatic commit operation after every successful query. Here’s what it looks like:

my $dbh = DBI->connect(‘DBI:Oracle:payroll’,

{AutoCommit => 1},

)

or die “Couldn’t connect to database: ” . DBI->errstr;

Automatic Error Handling

When you make the connect call, you can specify a RaiseErrors option that handles errors for you automatically. When an error occurs, DBI will abort your program instead of returning a failure code. If all you want is to abort the program on an error, this can be convenient:

my $dbh = DBI->connect(‘DBI:Oracle:payroll’,

{RaiseError => 1},

)

or die “Couldn’t connect to database: ” . DBI->errstr;

Don’t do This

People are always writing code like this:

while ($lastname = <>) {

my $sth = $dbh->prepare(“SELECT * FROM people

WHERE lastname = ‘$lastname'”);

$sth->execute();

# and so on …

}

Here we interpolated the value of $lastname directly into the SQL in the prepare call.

This is a bad thing to do for three reasons.

First, prepare calls can take a long time. The database server has to compile the SQL and figure out how it is going to run the query. If you have many similar queries, that is a waste of time.

Second, it will not work if $lastname contains a name like O’Malley or D’Amico or some other name with an ‘. The ‘ has a special meaning in SQL, and the database will not understand when you ask it to prepare a statement that looks like

SELECT * FROM people WHERE lastname = ‘O’Malley’

It will see that you have three ‘s and complain that you don’t have a fourth matching ‘ somewhere else.

Finally, if you’re going to be constructing your query based on a user input, as we did in the example program, it’s unsafe to simply interpolate the input directly into the query, because the user can construct a strange input in an attempt to trick your program into doing something it didn’t expect. For example, suppose the user enters the following bizarre value for $input:

x’ or lastname = lastname or lastname = ‘y

Now our query has become something very surprising:

SELECT * FROM people WHERE lastname = ‘x’

or lastname = lastname or lastname = ‘y’

Programming the Perl DBI Alligator Descartes and Tim Bunce

O’Reilly & Associates, Inc., United States

01 January 2000

Putting Complex Data into Flat Files (Programming the Perl DBI)

What is a Flat File, or Flat File Database?

DBI – The Database Interface for Perl 5 – The Perl Journal, Spring 1997

Transactions, Locking, and Isolation (Programming the Perl DBI)

The post Write up on Tech Geek History Perl DBI Database (revision 3) appeared first on DayDreamin’ Comics.

Born Terry Lee Goodkind on January 11, 1948 in Omaha, Nebraska, Goodkind suffered from dyslexia and initially had no interest in writing. In fact he had no formal education beyond high school. Instead, Goodkind worked various jobs including building cabinets and violins and as a wildlife artist, selling many of his painting to various galleries. Terry Goodkind also had a military background as a marine and enjoyed amateur and semi-professional car racing. It wasn’t until 1993 during the construction of a house for he and his wife Jeri in Maine, that he began writing his first novel in the Sword of Truth series, entitled Wizard’s First Rule. It sold for a record $275,000 at the time and  was subsequently published by Tor Books in 1994.

Over the next twenty-six years, Goodkind went on to publish an additional twenty-five novels and six novellas. His prolific works have sold over twenty-five million copies and been published in over twenty languages. Goodkind’s Sword of Truth series was eventually adapted into a television series called Legend of the Seeker that ran for two seasons. Rather than traditional fantasy, Goodkind’s works tended to focus on philosophical ideas and human themes.

From a personal standpoint this loss is like a roundhouse kick from a drow to my solar plexus. I can’t tell you how important and influential Terry Goodkind’s Sword of Truth series was for me when a friend introduced me to it as a freshman in college. Rarely have I ever felt a greater connection to the characters of Richard and Kahlan. Goodkind’s works as exciting, adventurous, and fantastical as they were, always had a strong moral code about them and spoke clearly to the human condition. They were philosophical, introspective, and in some ways spiritual. Terry Goodkind was very much an objectivist, the philosophy expounded by author Ayn Rand, and some of this ideology was prevalent in the series. While I personally consider objectivism to be a morally bankrupt philosophy, the characters of Richard and Kahlan and author Terry Goodkind most certainly were not.

Goodkind was also a living testament to the idea that it’s “never too late” to realize your dreams, as he didn’t publish his first novel until the age of forty-six. Other than his contemporaries George R.R. Martin, Robert Jordan, and Terry Brooks, I can’t think of a writer of fantasy that had more influence and meant as much to the genre as Terry Goodkind. A true giant of the fantasy world he will be sorely missed by me and his millions of fans.

Literature Review

Wizard’s First Rule is the beginning. One book. One Rule. Witness the birth of a legend.

1. Richard Cypher has always been an explorer.  From a young age, he enjoyed learning from his friend, Zedd, who is a cloud reader.  As an adult, he by guiding people through the forest.  He returns to his hometown, Heartland, when he hears of his father’s murder.  His older brother, Michael, tells him not to investigate.  Richard goes to his father’s house anyway and sees a blue clay jar he made as a child.  Usually, there is a vine in the jar (one that he hasn’t seen anywhere else).  Richard decides to go to the woods and look for it to try to find clues about the murder.  When he finds it in the forest,  it is strangling a tree.  He goes to tear it down, but it stabs him with a thorn.  When he tries to get the thorn out, it burrows in deeper.  He curses himself for forgetting his knife.
   
   2. Realizing that he is near the boundary, Richard starts heading back towards town for the celebration his politician brother is throwing.  He will now be in charge of all the Midlands, rather than just Heartland.  Richard sees a girl in a dress too fancy to be traveling through the woods in.  He notices that there are four men behind her that seem to be stalking her.  He decides to take a path that will cut her off and tell her what is going on.  As he waits for her to arrive, he gets nervous that he imagined the danger in the situation.  When she arrives, he tells her about the men.  She asks if he will help her.  He agrees and tells her they should head down the narrow path to get away from them.  She goes with him.  Eventually, they come to some rocks and start climbing up.  When they reach the top of the cliff, two of the men come from the other side of the path.  Two more drop in behind them.  Richard prepares himself to stand with the girl, though the men offered to let him go.  When they attack her and Richard, he feels thunder without the sound.  As one of the men charges at them, Richard kicks him into the last remaining one who is already bloody and they fall off the cliff together.  Richard grabs the next one and throws him into the leader and they both go over the cliff as well.  She explains that the men were a Quad, a team of assassins.  They introduce themselves to each other and he learns her name is Kahlan Amnell.  She says that she and the Quad are both from the Midlands, the magical land beyond the boundary.
   
   3. Richard takes Kahlan to his brother’s house for the festival.  When they arrive, his friend Chase comes to talk to them.  Richard introduces them.  He then asks his friend, a Boundary Warden, what he was doing there.  Chase explains that Michael called in all the Wardens, saying it is for effect.  Richard quietly tells him about the men they ran into in the forest, but doesn’t say how they defeated them.  Chase suggests that people are coming from the Midlands, over the boundary.  Richard starts to wonder if there is truth to what Kahlan said and remembers hearing people saying that others were snatched from air by horrible creatures.  Chase tells him that Zedd wants to see him as soon as possible.  Richard, looking up and seeing the snake shaped cloud he noticed earlier, realizes he needs to talk to Zedd too.  Kahlan gets acquainted with the Westland customs.  She notices two people watching Richard.  He points out that they are his brother’s advisors.  She explains that the Quad should not have been able to follow her.  She had the help of five wizards to cross the boundary.  The men shouldn’t have been able to follow.  He plans for them to stay at his house for a night and then find Zedd to get some answers.  Michael appears and gives his speech about accepting the position of First Counselor of Westland.  He says that he is ready to lead them in a war against traitors and conspirators.  Richard is not sure what he is talking about.  Michael continues on to talking about the day when the boundary no longer exists.  He wants to unite Westland, Midland, and D’Hara.  He wants to stop those that want to start an unnecessary war with the people over the boundary.  He continues on the talk about their mother’s death by fire.  He goes to Richard afterwards to explain that he wants the people to take on the future with excitement, and that was the spark he needed in them.  Richard asks about their father’s murder.  Michael indicates that their fool father took something and someone came looking for it.  Richard realizes that Michael did all that he could.  Richard introduces Kahlan, who explains that she is the reason Richard was late and a mess.  Suddenly, Michael molests Kahlan.  She takes her fingernail and cuts his chest to get him to release her and walks away.  Richard, disgusted with his brother, follows Kahlan.  Richard follows.
   
   4. Kahlan tells Richard people are after her because they fear truth.  She says that she cannot tell him everything without increasing the danger to both of them.  He asks her to tell him someday because he is a seeker or truth.  She asks why he calls himself that and he explains Zedd had always called him that since childhood.  She tells him that Darken Rahl’s followers are the ones that are after her, but refuses to give any more information.  Richard thinks about the things his father trusted to him, but he forgot at the house that morning.  There was a book and a tooth he strung and usually wore around his neck.  As they approach Richard’s house, he starts to feel uneasy.  When they arrive, the house is ransacked.  Kahlan points out that the person that did it could still be inside.  Richard sneaks around back to get the tooth that belongs to the keeper of the secret book.  He grabs it through the window.  He hears the chair squeak.  He quickly grabs the tooth, puts the string around his neck, and leads Kahlan away from there.  As they make their way towards Zedd’s, Richard realizes that the same person that killed his father could be the one that was in his house.  Kahlan asks what he went in for, but he refuses to answer.  He remembers that the books were torn apart and wonders if someone knows about the secret book.  Suddenly, Kahlan takes Richard to the ground.  She tells him not to move or else they are dead.  She points to the danger and he sees a boar with glowing green eyes.  They watch it charge and kill a rabbit and then extend its wings and fly away.  Kahlan tells him it is a Garr that uses blood-flies to hunt, which had been biting them.  
   
   5. As he builds a fire, she tells him that it came from D’Hara.  Richard is shocked to hear that and she explains that the other boundary (between D’Hara and the Midlands) is already down.  Kahlan explains the history of the three lands.  The D’Haran ruler, Panis Rahl, started swallowing up territories and took over all D’Hara.  The Midlands join together to be free lands and fight off the D’Harans.  Rahl turned to magic, the worst being the shadow people (which killed with just a touch).  A great wizard joined the Midland’s cause and created horns that blew away the shadow people.  They knew they couldn’t bet the the D’Harans, but needed to do something.  The boundaries were created, the Westlands were set aside for those that wanted to live without magic.  The boundaries were conjured by magic from the underworld.  To walk through the boundary is entering the land of the dead.  Kahlan was able to cross only with Magic.  He asks her about going through the underworld and she is taken mentally back there.  Richard brings her back and asks her to continue the story.  She tells him how the great wizard defeated Panis Rahl, but not before he killed the wizards wife and daughter.  The Council in the Midlands became corrupt and the great wizard left them, casting a wizard’s web to keep anyone from remembering who he is.  Suddenly, people in the Midlands started talking about Panis Rahl’s son, Darken Rahl, and how he was bringing peace.  The D’Harans came into the Midlands when the boundary failed and started killing those that didn’t agree with him.  He banned magic and taken over much of the Midland.  Five of six of the great wizard’s students saw the error of their ways and have sent Kahlan to ask him to make the appointment only he can make.  If he says no, the wizards do not have the power to force him, but Kahlan does.  Richard wants to know more, but decides not to press.  Kahlan pulls out a night whisp named Char, which can only be seen at night.  She explains that the whisp helped her through the boundary and will now die from being away from her own land too long.  Char explains that Darken Rahl will continue to come after Richard and Kahlan until they are both dead, but can’t tell him why.  Char speaks to Kahlan alone.  She asks why Kahlan hasn’t told Richard what she is.  Kahlan explains that she doesn’t want to loose his friendship, as she has never had anyone treat her that way.  She promises to tell him before anyone else can.  Char asks Kahlan to use her power by touch to let her die feeling love.  Kahlan complies and there is the thunder without sound again.
   
   6. The next morning, Kahlan makes an offering to their dead mothers for good fortune on their journey.  Richard hands her a red apple and she thinks that he is trying to poison her (any red fruit is poison in the Midlands).  Richard realizes that the fever from the thorn is getting worse and they make their way to Zedd’s house.  They find him out back, naked and watching the sky on his “cloud rock.”  Richard helps him get dressed and points out that they have company.  Zedd goes on about the cloud, which Richard guesses is from D’Hara.  Zedd finally introduces himself to Kahlan and realizes what she is.  Richard explains that she saved him multiple times.  Zedd accepts her.  Kahlan and Richard tell Zedd about their journey so far.  Richard shows Zedd the vine.  Zedd tells them that when the boxes of Orden are in play the snake vine will grow.
   
   7. Richard wakes up some time later after the fever makes him pass out.  He suddenly remembers everything, including the secret book, The Book of Counted Shadows, that his father had chosen him to keep.  Richard swore to keep it secret until the keeper came back for it.  Richard would memorize the book piece by piece until he memorized it all and it was burned.  Zedd is out looking for the root he needs to cure Richard.  Richard asks Kahlan not to hurt Zedd or ask for his help, no matter what, before falling asleep again.  Zedd returns and makes the medicine.  Together they administer the cure to Richard.  Kahlan explains that she needs to find the great wizard to appoint a Seeker.  Zedd explains that a seeker cannot be simply appointed, but a wizard just points out a true seeker and makes the title official.  The Council members previous made the appointments, but it was out of greed for the post, rather than presenting an actual Seeker.  He doesn’t believe that even a true Seeker would be able to defeat Darken Rahl alone.  Kahlan explains that she plans to stand the Seeker that is found.  He tells her not to claim Richard or ever to touch him with her power, and she agrees.  He tells Kahlan to get some sleep and leaves the house.  He runs into a Gar and threatens to kill him unless he goes back to D’Hara and kills the Quad hunting them on the way.  The Gar agrees and leaves.  Zedd continues on to a rock.  He uses magic to bring forth apparitions of his parents.  They talk him into doing his duty rather than giving into his grudges.  They tell him that Darken Rahl only two of the three boxes of Orden and still doesn’t have the Book of Counted Shadows.  Darken Rahl walks the underworld, which is how he was able to undo the boundary.  They warn him not to go through the boundary.  He asks them how to get to the Midlands, but they do not know.
   
   8. When he wakes up again, feeling much better, Richard approaches Zedd and Kahlan.  He tells Zedd that he already knows that he is the great wizard by what Kahlan has told him.  Zedd is impressed with his power of deduction and goes to get the sword of the Seeker.  Kahlan begs him not to name Richard as the Seeker, but Zedd explains that he doesn’t have a choice; anyone else would not be a true Seeker.  the great wizard names Richard as the Seeker, and hands him the sword, which gives him a fierce fury for truth.  but Richard doesn’t know what that means.
   
   9. Zedd explains that Darken Rahl has started collecting the boxes or Orden.  Once the boxes are put in play, that person has one year to open the first box.  If they fail to open the box, or open the wrong box, the consequences are dire.  One wrong box would kill the person that opens it.  The other wrong box would kill everything living.  If Rahl were to open the correct box, he would gain the power or Orden.  Richard already knows all of this information because it was in the Book of Counted Shadows.  He realizes that must be why Rahl wants to kill him.  Zedd points out that Rahl only has two of the boxes and must be very confident for putting them in play before having all three.  Kahlan tells Richard that Darken Rahl reads the future by using people’s entrails.  He grabs the sword and becomes angry, vowing to kill Darken Rahl.  Zedd calms him down, but is impressed that Richard is now able to call upon his anger, saying that he has passed the final test in order to be a true Seeker.
   
   10. To make a point, Zedd temporarily names Kahlan as Seeker.  Zedd tells Kahlan that one of two trees was planted by Darken Rahl to track him.  He reminds her that Darken Rahl killed her sister and is trying to kill all of them.  She cuts through the bigger of the trees.  He renames Richard as Seeker and tells him to cut down the smaller tree.  Zedd tells Richard to cut down the second tree.  He is unable to because he believes that the tree is not evil.  The sword will only cut down what the Seeker believes to be evil, so he’d better be sure before he strikes.  Zedd continues to explains that there is a price to be paid for each kill with the Sword of Truth; His anger is a shield from the pain the sword will inflict with each use.  Zedd then explains that Richard cannot use the Sword of Truth against Darken Rahl.  Richard understands.  They start to make preparations to leave for the Midlands.  Richard thinks he has a plan to go through an opening between the two worlds; the one his dad used to bring the Book of Counted Shadows to Westland.  A mob comes along, angry about Zedd using magic.  Richard points out that Zedd has helped them, but the villagers refuse to listen.  Zedd confuses them by pointing out how they have come to attack a powerful wizard with just make-shift weapons.  They go to leave, but Zedd makes them think he took their genitals to teach them a lesson.  The apologize, and the great wizard lets them leave.  Richard realizes that Rahl is trying to lead them into a false sense of security and they leave quickly.
   
   11. A pair of gars attack.  One tackles Zedd and the other Richard.  Kahlan helps Richard and together they are able to defeat it.  Zedd’s house is on fire in the background as they hope that Zedd has not seen their battle with the gars.  They make it to Chase’s house and he welcomes them in.  They tell him what has happened and Richard asks him where they can pass through the boundary.  Chase hesitates.  Richard shows him his sword and Chase knows that Richard is the Seeker.  He tells Richard that the Boundary Wardens real job is to make sure that the creatures in the boundary aren’t able to feed enough to come out and hurt the people in Westland.  He and his wife, Emma, help them prepare to leave.
   
   12. The next morning they take off on horses. Chase notices wolves eating and goes to make sure that they are safe.  Zedd and Kahlan encourage Richard to move on rather than going back to help his friend.  They come to a bridge, but Richard isn’t sure that it will be able to hold the weight of the horses.  Zedd sends his horse across and they are able to go over the water.  Once they get to the other side, Zedd destroys the bridge.  They continue on.  Richard thinks that he hears something and draws his sword.  Just as he is about to strike, he realizes Chase has caught up with them.  He explains that he has been keeping the things from inside the boundary from getting them.  There are heart-hounds, created when the boundaries were erected.  They are able to go in and out, but can’t usually stray too far.  Kahlan tells them that they are all over the Midlands from the other boundary coming down.  They are free and crazed with lack of purpose.
   
   13. Richard takes the first watch over the group.  Kahlan comes to talk to him before going to sleep.  He realizes that he loves her and asks her if she has anyone.  She says that she has obligations.  Richard points out that she also has an obligation to lead her through the Midlands.  Zedd comes to talk to him before his watch starts.  He tells his friend how he feels about Kahlan.  The wizard explains that Richard cannot be with her, but says she will have to tell him why.  Richard asks him about Kahlan threatening him over the red apple.  The wizard tells him that, like Kahlan, he will have to be prepared to kill someone if they threaten their cause.  He explains that Panis Rahl made all of the red fruit poisonous to get to the people by killing their children.  Zedd says he would need to study the spell to find out how to do it.  Richard comes up with an idea to confuse the spell Darken Rahl has on the cloud that has been following him by making it attract other clouds.  Zedd pulls out his wizard’s rock (the cloud rock).  Richard goes on the rock and feels at one with everything.  Zedd points out that Richard could make a great wizard, but Richard turns him down.
   
   14. The next morning, Chase leads them.  He shows him the damage the heart hounds can do by putting a long stick into the boundary.  He takes Richard to the boundary.  Richard sees his father calling to him, telling him to join him in the space of underworld.  Chase holds him back and explains that he had to prepare Richard for what he would see and how easily he could stray into the underworld.  Zedd predicts the boundary will last a few weeks.  Richard explains that he doesn’t think it was really his father that he saw, but Darken Rahl’s disciples trying to lure them into the underworld.  Suddenly, Dark appendages come out of nowhere.  Chase and Zedd are knocked down.  It grabs Kahlan and tries to drag her in.  Richard slashes at the thing, even though Kahlan begs him not to because she thinks it is his sister.  He grabs Kahlan and they get away and tells her to stay while he goes to get the others.  He finds Zedd and the tentacle comes after him again.  His rage explodes and cuts it off.  Richard is able to get his friends.  He loads them on their horses.  Richard knows they have to get to the swamp Chase had talked about to escape the oncoming heart hounds.
   
   15. Richard gets them to the swamp.  Kahlan points out that the monster that attacked from the boundary couldn’t seem to see him.  Richard wonders if it is because of the sword.  He also realizes that when he killed the things from the boundary, he didn’t feel the weight of their deaths, which is supposed to be the price for using the sword.  The next day, they travel further to town.  They go to an inn and talk to the keeper, Bill.  The crowd inside is rough and tries to indicate they want Kahlan.  Richard starts to go for his sword, but Kahlan solves the issue by saying that Richard is his whore and she will contact them if she needs more.  The proprietor appreciates that she stopped a fight that could destroy his business.  Bill gives them a room and offers his son to help get Zedd and Chase inside through the back.  The inn keeper has a new respect for them once he finds out Chase is with them.  Richard tells him that he is the first councilor’s brother and that they are on an official mission of Westland.  Before Bill leaves, Richard warns him that the boundary is failing.
   
   16. Bill wakes them up the next morning because half a quad arrived at the inn.  His customers had taken care of one and the other flees.  He tells Bill that his brother will cover all of the charges.  They make their way to the bone woman’s house.  They meet AdieAdie and she tells them to bring their friends.  She explains to them the strength of the tongue compared to the teeth (sword).  Adie knew Kahlan’s mother and gave her a bone necklace to protect her unborn child.  She uses a bone tool to wake Zedd and Chase and make them drink.  Richard is so relieved that his friends have received help that he offers to make Adie spiced soup.  He leaves the women to talk in their native language and chops wood for her.  Adie explains that their friends will need several days to recover with constant watch.  Richard determines that they can’t wait and risk Darken Rahl getting the last box or Orden.
   
   17.  Adie explains that the bones of the dead creatures that live in the boundary will protect them enough if they are careful.  She tells them that she became overconfident and decided to take a nap in the pass one day when she had gone to far.  When she woke up, a gripper was on her ankle and she had to cut off her foot to get it off.  They also have to worry about going through the rock that will take them through the narrows.    She gives Richard a night stone that will light their way when it is too dark.  It will also help Zedd find them when he is well.  Adie gives Kahlan a bone necklace like she gave her mother.  Richard asks if he should have a bone as well, but Adie points out the tooth under his shirt that already hides him from the boundary creatures.  They sleep that night and prepare to leave in the morning.Richard leaves orders for Chase to stand guard against Darken Rahl, no matter what his brother says.  
   
   18. Richard and Kahlan take off down the path through the pass. He starts thinking about how to stop Darken Rahl from finding the last box or using them.  He recites the Book of Counted Shadows to himself to find out what he can do, but it isn’t as useful as he’d hoped.  As it gets dark, shadows start to surround them.  Richard wonders if they are the shadow people Panis Rahl used in the last war.  They carefully make their way, making sure not to touch the still shadows incase they have the deadly touch.  as they reach the narrows, they are surrounded by grippers on the rocks around them.  Richard pulls Kahlan through  the opening and they go on.  
   
   19. Meanwhile, Darken Rahl is in the tomb of his father.  His guard, Demmin Nass, brings him a boy meant to be a sacrifice.  Rahl points out that Devon has ruined his spirit guide before.  When he tried to blame it on someone else, he submitted to the mother confessor, Kahlan, to prove his innocence.  Demmin changes the subject and asks what to do about Queen Melina’s demands.  Rahl tells him to give in to all of her demands for the box and he will deal with her himself if she gives him any issues.  As he is about to leave the tomb, a rose petal falls.  Rahl sticks it to one of the guards forehead, wondering why they are disrespecting his requests to keep his father’s tomb neat.  Darken Rahl knows that Kahlan has found the Great Wizard who has named a seeker.  He thinks that Kahlan must have used her power to force him.  Rather than sending another Quad, Rahl plans to make Kahlan his to make an heir.  Rahl approaches the boy Demmin brought him, Carl, now buried in sand in a temple-like setting.  The boy says that he want to go home.  Darken Rahl calms him down by saying he isn’t going to hurt him.  He needs the boy for a ceremony because he is said to be brave, strong, and smart.  He repeats that he wants to go home, and Rahl relates to him by telling him about his parents and telling him he needs Carl to help their people.  When Rahl goes to leave, Carl asks him to stay the night with him.  
   
   20. Richard and Kahlan make their ways into the Narrows.  Kahlan leads the way with Richard holding her hand and walking backwards to guard their backs.  He thinks about pulling out the sword out, but he has to much fear and the righteous rage of the seeker goes away.  Suddenly, the shadows start whispering to him.  He starts to feel comforted by them and wants to go to them and their promises of living how it used to be.  He wants to take Kahlan with him, but he can’t find her.  He suddenly realizes what is happening, but Kahlan is no where to be seen.  He pulls out the sword and the rage takes over.  He cuts apart the shadows as they come at him.  The seem to keep coming.  He gets away and makes it back onto the path.  Richard’s father appears and tells him that Richard can see Kahlan if he comes with him.  Richard stabs the thing that looks like his father.  He realizes that he is a tool of death and he must kill Darken Rahl.  Carl wakes up to find Darken Rahl still with him.  He talks to the boy, convincing him that his parents don’t love him.  He needs to have the boys completely loyalty to travel the underworld.  Demmin tells Rahl that the tracer cloud over Richard is hidden.  Rahl explains that he can always find Richard because he wears the tooth of the monster that originally guarded the Book of Counted Shadows.  Demmin leaves to visit the queen and Rahl tells him to stop by to see the dragon on his way.  
   
   21. Richard decides that with Kahlan alone, she would have no need to continue on to the Midlands.  He decides to go back and look for her.  In the Narrows, he finds her tracks going back the way they had come.  He pulls out the Sword of Truth and uses the strength of rage to go after her.  He sees Kahlan and the last man from the Quad.  He rushes at the man and kills him with the sword.  Suddenly, he has to pay the price of the sword.  He feels the incredible guilt and Kahlan comforts him.  She apologizes that he had to kill a man because of her.  Zedd had warned Kahlan that the magic of the sword would be the worst with his first use.  They move as quickly as possible back through the narrows and make it to the Wilds.  They are finally in the Midlands and able to rest.  He explains that they need to find someone with magic that can help them find the box.  Kahlan suggests they see the mud people, who are afraid of her.  She warns him that he will have to do what she tells him to get them to help them.
   
   22.  Zedd wakes up earlier than Adie had expected.  She tells him about helping Richard and Kahlan.  He checks the night stone and is able to tell that they made it to the Midlands.  Zedd will not be joining them right away because he has other things to attend to.  Adie asks him to stay the night and he obliges.  Richard and Kahlan arrive at the village of the mud people.  Kahlan starts to talk to one, Savidlin, and he slaps her.  She slaps him back and Richard starts to pull out his sword.  Kahlan explains that is how they show respect.  The man punches Richard and he punches back hard, pushing the mud person back.  They welcome the visitors and take them into the village
   
   23. Once they reach the village of the mud people, Richard and Kahlan are brought before the elders.  Light slaps are given to show respect and they begin to talk.  The elders think that Richard is a spirit man that brought them the rain.  Kahlan suggests they use deception to get them to call a gathering to find the box.  Richard refuses to lie to them and asks that they consider his request, judging him while he stays with them.  After everyone leaves, Richard notices that the Bird Man is still watching them.  Realizing that this is where the real authority lies, he goes to talk to him.  The Bird Man tells him that if he can convince the mud people to accept him as one of them, then they will call a gathering for him.  Richard asks Kahlan if they have any other options.  There are some alternatives available, but Richard thinks it over and decides to show the mud people how to make roofs that don’t leak (as theirs are currently made of grass).  The Bird Man says he is not sure how the people will respond, but he is free to try.
   
   24. Rahl has spent days taking care of Carl, who has become completely loyal.  He thinks about the girl that he killed the night before to keep his thoughts calm.  When Carl insists that he stay with Rahl to protect him, Rahl says that they will be able to perform the ceremony that night, after the feeding.  Rahl retrieves a crucible and chants.  He feeds the molten lead to Carl through his feeding tube.  He sends the boys spirit to the underworld and instructs Demmin to put the body on the alter.  He continues to chant as he touches the scars left on his genitals from the wizard’s fire that killed his father.  He removes the heart, brain and testicles, mashing them into a paste from the boy and draws runes on the sand where Carl had been.  He eats the mixture and calls to the boy, who rises as a shinga (underworld beast).  The shinga comes and takes Rahl to the underworld.
   
   25. As Richard shows the mud people how to build clay tile roofs, Kahlan is mostly left alone because most people in the Midlands are afraid of her.  She starts thinking about when her mother died and her best friend’s mother took her in.  Dennee and Kahlan grew up like sister, but Kahlan was always stronger and looking out for her.  While returning from a trip that Kahlan had volunteered to take (knowing Dennee was not fond of confessions), she finds her sister dying in the road.  She promises Dennee that she will kill Darken Rahl.  When the roof on the town prayer house is complete, everyone gathers.  The elders are not impressed and order it taken down and restored to how their ancestors had built it, refusing to be influenced by outsiders.  Richard offers them his sword so that they can give their people the favor of quick deaths.  They refuse and refuse to yield on their decision.  Richard and Kahlan go to pack their things and they find that the night stone is missing.  They think Savidlin’s son, Siddin, has it.
   
   26. They race to find Siddin and Richard makes it to him.  The child just got the night stone out of the pouch and the shadow people start to surround him.  The mud people start shooting arrows at the shadows.  Kahlan has to instruct the Bird Man to keep them from attacking, informing them that touching the shadows would kill them.  One of the elders demands that they kill Kahlan for disrespecting their ancestors’ spirits.  He attacks her and she goes to use her power on her.  She hesitates, not wanting Richard to see what she can do, and gets cut.  She follows through the second time and shoves the man into the ring of Shadows around Richard and Siddin.  She uses the disruption to get to them.  She has to rip the stone from Siddin’s hand and puts it back in the pouch.  The shadows start to slowly drift back to the underworld.  Siddin’s mother, Weselan, is very appreciative and hugs and kisses Kahlan.  Savidlin is appointed an elder and demands that they make Richard and Kahlan mud people.  The elders offer their lives to Richard for being foolish, but he returns their lives and honor to them when the sword won’t cut them.  Richard and Kahlan are made mud people and the Bird Man points out that she will have to make sure that Richard follows their way of life. The Bird Man sends Kahlan to a healer, Nissel, to close up her cut.  She warns the healer that she cannot be made drowsy or else she could accidentally unleash her power.  Once they are both cleaned up, Kahlan and Richard meet up again.
   
   27.   That night, there is a celebration for the gathering that will take place the next night.  There are several girls following Richard around and he asks Kahlan why.  She tells them that he will have to pick one and impregnate her.  He seems delighted by the idea and Kahlan is furious.  He chooses the Bird Man’s niece and asks Kahlan to get him an apple from his bag.  She does and he starts eating it.  Everyone stares at him in surprise as he eats the red fruit.  He explains that everyone in the Westlands eats them, but there is a possibility that they could make his seed poisonous to Midlanders.  The Bird Man agrees to let him skip the tradition.  Richard retires for the night and Kahlan feels awful about not telling him what was expected of him.  After talking to the Bird Man (who gives her a bird whistle for Richard), she goes to find Richard and he forgives her.  Suddenly, she starts to act on her feelings for him.  Kahlan starts to think it wouldn’t be such a bad thing if she took him until he says that she has to tell him what she is before they can go any further.  Kahlan stops and is horrified at what she almost did.  She asks him to hold her.
   
   28. Richard and Kahlan wake up and get dressed.  The Bird Man spends the day trying to teach Richard how to use the whistle, but he is not good at it.  When it is time to start the gathering, the Bird Man tries to explain that women are not allowed because they have to meet the ancestor spirits as they are.  Kahlan explains to Richard that the spirits will kill anyone that they don’t believe have the right intensions.  She is worried that if she isn’t there to translate, Richard will be killed.  She explains what must happen for her to go with him and he agrees.  They enter the spirit house one by one and strip and are painted with mud.  A basket of frogs are passed around to be rubbed on the chest.  Kahlan is worried that she could accidentally use her power in the altered state the frogs will cause, but the Bird Man convinces her that she will be killed if she doesn’t do it.  When the spirits arrive, Kahlan worries because it looks like Richard is being hurt.  They ask him a series of questions before telling him the name of a witch woman that knows who has the last box.  The spirits tell them that Darken Rahl is there and killing people, but if Richard goes out to face him, he will die and fail.  When the gathering ends and they go out into the village, one man comes to blame Richard for what has happened.  Many people were killed and they were warned not to use fire anymore.  Richard says that they can blame him if it helps, but he will not stop until either he or Darken Rahl is dead.  Savidlin volunteers to escort them to the edge of their lands
   
   29. Princess Violet abuses Rachel and bosses her around.  Rachel does her hair and helps her put on jewelry.  When the princess grabs a box she’s not supposed to touch, Rachel freaks out (even though magic will not let it leave the room).  Princess Violet gives Rachel the key to the room and tells her to clean up.  She saves putting the queen’s box away for last.  As she is putting it back, she notices that someone is in the room with her.  She meets Wizard Giller, who is nice to her.  He heals her cheeks where the princess has slapped her.  He gives her a trouble-doll and says that he will hide it for her until she is sent outside to sleep by the princess.  Rachel makes her way down to the kitchen.  She waits for the cooks to stop arguing, but they end up spilling the crock.  After they clean it up, she tells them that the princess sent her to scold them.  They give her some meat and pie before sending her on with the promise not to tell anyone that it happened.  That night at dinner, a fool is brought in for the royals’ entertainment.  The queen claims that he is a traitor to the common good.  He argues that they are taking all of his food.  They are not allowed to use fire and so his family is also freezing.  The queen decides to let her daughter decide his fate.  Princess Violet orders that the man is beheaded.  After everyone else has left, Rachel tells the princess that she was mean to have the man killed.  She sends Rachel out of the castle for the night.  Rachel acts sad, makes her way through the garden to get the hidden doll, and leaves the castle for her hiding place.  Zedd finds D’Haran guards and looks out on the sleeping city.  There are troops everywhere.  He knows that Darken Rahl had been there and had taken something.
   
   30. After Kahlan explains the different types of dragons in the Midlands, they are approached by Old John.  He says that he comes from the old wizard that needs help.  After traveling for a while, Richard is winded.  The man seems to be going strong and wonders if something is wrong with him.  Richard asks him for Zedd’s last name, but Old John cannot answer.  It changes into a Kalthrap and attacks.  A smaller thing attacks and they roll away, the Kalthrap going into the woods alone.  Richard threatens the smaller thing until it explains that he and his mistress, Shota, were following them.  Shota has Kahlan.  Richard decides to take the thing, Samuel, to go after Shota.  He takes him to Agaden Reach.  Kahlan is there and there are snakes covering her.  Richard promises to save her, but Kahlan tells him that if she even moves, she would be dead.  She tries to convince him to leave her and save himself.  Richard still needs to find out about the box, so he cannot leave.  Rachel tells her doll all of her worries.  When she starts to wonder what to do, the doll tells her to help Giller.
   
   31. There is a voice and Richard sees his mother.  She embraces him and he wonders if it really is her.  Suddenly, Zedd is behind him telling Richard to get away from her.  Richard realizes the woman is Shota.  Zedd starts to fight the witch woman, but Richard tries to stop him because she knows where the box is.  Shota freezes Zedd.  She offers Richard a wish for coming to her rescue (even though it is not necessary).  Richard wishes for Shota to let Kahlan go without hurting or killing her after the witch woman agrees that he doesn’t have to use the wish to get an answer to his question answered.  She explains that Zedd was not real, but an illusion.  Samuel is the last seeker, who Zedd took the sword from.  Richard has some insight to his future after using the magic.  She warns him that Zedd will actually use the wizard’s fire against him someday.  She also tells Richard that the sword cannot protect him against Kahlan’s magic, which will happen before Rahl opens the boxes.  Kahlan begs for death.  Shota cannot provide it because it would be against Richard’s wish.  Kahlan tries to kill herself, but Richard stops her.  Shota tells him that Queen Melina has the last box.  Before she leaves, the witch woman warns him never to come back to her.
   
   32. Rachel waits for Giller to approach her as the doll, Sarah, told her to.  He tells her that they will runaway together soon, but he has a few preparations to make first.  The queen approaches and she hides behind his robes so that no one is suspicious.  The queen mentions that Darken Rahl is coming the next day and heads off with her dog.  The wizard seems worried.  He tells her that they will have to run away carefully before the next day or they will both be killed.  Giller tells her to act normal and to get a big loaf of bread from the kitchen and to meet him in the jewel room.  On her way, Rachel runs into Princess Violet.  Rachel has to talk her into letting her scold the cooks.  Princess Violet decides to go to the jewel room, and Rachel tries to convince her not to, but is unsuccessful.  Rachel decides to stick to the wizard’s plan.  Richard and Kahlan come across a pack of gars.  Kahlan has been extremely withdrawn and is being careless enough to get herself killed.  Richard has to stop her several times.  Rachel makes it to the Jewel room with the loaf of bread and waits for Giller.  He brings in a fake copy of the box and puts the real one in Rachel’s hollowed out loaf of bread.  He explains that Darken Rahl is evil and wants the box.  He needs her to take the box out of the castle and hide.  Richard finally gives up on Kahlan and gives her permission to go kill herself.  He explains that he will not forgive her.  He thinks that if Kahlan and Zedd are going to fight against him, maybe he will be the one to betray them.  Kahlan refuses to be the one to destroy Richard, so she goes, wishing he wouldn’t remember her like that and walks away.
   
   33.  Darken Rahl comes to the castle early.  Queen Melina is surprised but is ready to make their treaty and give him the box.  She sends Giller for it.  When he hands it over, Rahl gives it to one of his men and he smashes it.  They know it was a fake.  Giller tries to tell them it must have been a forgery from the beginning, but Rahl is not convinced and they take him away.  Rachel sneaks around to get the loaf of bread and get out of the castle.  She just makes it out when she realizes that she forgot her doll.  She goes back and tells the guards that she forgot something.  She overhears two people talking about how the D’Harans are going to search the entire castle.  Rachel hurries to the princess room and opens up her sleeping box.  Her doll, Sarah, is not there.  Princess Violet reveals that she found the doll and threatens to through it in the fire.  Rachel pulls out the fire stick Giller gave her and starts setting small fires.  she holds the stick up to the princess and threatens to burn her.  Violet gives her the doll, and Rachel makes her get in the sleeping box.  Rachel hurries back out with some food, the loaf of bread (with the box inside), and her doll.  A D’Haran guard tries to stop her from leaving, but the Queens guard convince him that she is harmless after they search her.  She is let out and starts running away towards the west.  Sarah tells Rachel to hide like Giller told her and that he loves her.  Sparks and fire ignite a distance away towards the castle.  The doll doesn’t talk anymore.
   
   34.  Richard wakes up in the morning and finds that Kahlan has been watching over him all night.  She is ready to tell him her secret.  She is the Mother Confessor, a title given due to her significant amount of power.  She must constantly contain the magic or else it would spill into whoever she touches.  The power would find even the smallest spark of love and use it to make the person completely devoted to her.  The confessed is left a shell of their former self and practically a slave to the confessor.  It takes a while for the power to return.  Depending on the confessor, it could be up to days.  Kahlan’s power returns within a few hours.  She explains that most are afraid of her because she is past the age where she should have taken a mate.  People are afraid that a confessor will take them or someone they love.  Richard is completely destroyed, but points out that she keeps mentioning that confessors are women.  She tells him that male confessors don’t have to wait for their power to return.  If a boy is born, the father is forced to kill the child.  Richard decides that after he defeats Rahl, he will go home.
   
   35.  Back on their way, Richard and Kahlan stop in Horner’s Mill, which is mostly destroyed.  Kahlan goes to see about getting horses, but gets stories of what happened instead.  The D’Haran People’s Peace Army came in and started praising Darken Rahl.  Lumber orders started coming in and the town prospered.  People came in claiming to be Westlanders and reeked havoc, saying they will kill anyone who doesn’t oppose Darken Rahl.  the Westlanders came back to execute any men that did not flee the town.  The woman and little girls were raped.  Rachel’s hiding space is invaded by Richard and Kahlan.  As soon as she sees Kahlan’s long hair, she is afraid of her (only woman of stature have long hair).  When she overhears Kahlan say she wants to kill Giller, Rachel’s mind is made up.  However, she is fond of Richard, who offers to share their fish and cut her hair straight.  Rachel yells at Kahlan when she starts to reach for the loaf of bread.  Kahlan apologizes and wraps herself and the girl in a blanket.  Richard wakes Kahlan up in the middle of the night.  Rachel waits until Richard is asleep before she takes off.
   
   36.  The next morning, they wake up to find Rachel gone.  They don’t have time to go after her and they continue on to Tamarang, where Queen Melina lives.  Zedd comes to join them.  Richard threaten him at first, wanting to make sure it is him.  As they catch up on events of their separation, Zedd points out that the wizard’s first rule is that people are stupid.  They will believe what they want.  Zedd is the keeper of the Book of Counted Shadows, but he never opened it because he didn’t want the information to spread.  He kept it in his wizard’s keep, but it was stolen.  He wants to destroy the book.  Richard knows that he is the book and Kahlan is the only way to verify that.  He decides not to tell Zedd the truth to spare Kahlan.  Richard takes the first watch when they go to sleep.  He attaches the tooth he wears around his neck to a log and tosses it into the woods.  When Zedd comes to relieve him in the night, and Richard finally lets out his feelings for Kahlan and how they cannot be together.  He asks about Samuel and the fate that lies ahead for him.  Zedd tells him that it will not happen to the seeker that can turn the sword white.  He promises to help Richard figure out how to do it once they defeat Darken Rahl.  When Kahlan goes to take over watch for Zedd, she asks him if there is anyway to release her from being a confessor.  The only way is death.  She asks him to tell her the prophecy about her, but he refuses.
   
   37.  When Richard wakes up, he uses Zedd’s own rule against him to figure out that Kahlan told him what Shota said.  Zedd shows Richard how they all use additive magic, but cannot use magic to destroy.  He can grow a beard, but he has to shave it off when he wants to get rid of it.  All magic from that world is additive, but magic from the underworld is subtractive magic.  While they continue to travel, they give him all the details of what happened (except what happened between them at the spirit house).  Rachel keeps going, she sees something fly by in the sky.  She settles down for the night and then is bit by flies.  She tries to get away from them and runs into a gar.
   
   38.  Richard and Kahlan argue over the strategy they should use to get the box from Queen Melina.  A wolf arrives and knocks Richard over to save him from a heart-hound.  Kahlan introduces them to Brophy, who has explained that Richard has been leaving food for him since he started following them.  As a human, the wolf was accused of killing children and asked for a confession to prove his innocence.  Giller turned him into a wolf to dull the magic and allow him to live a relatively normal life.  
   
   39.  Tamarang is overflowing with people seeking protection.  Kahlan arrives at the door with Zedd and Richard on either side of her.  The guards are reluctant to let her in, but denying a confessor entrance would admit guilt.  They are brought before the queen, who introduces her daughter Violet.  When the princess insults Kahlan, Richard takes out his sword and threatens to cut out her tongue.  Kahlan tells Queen Melina that she is going to inspect her dungeon.  She finds all the innocent men in a cell.  She asks if any of them would be willing to confess.  When they all volunteer, she points out that she has only had three people ask for it in her life.  She frees them all without using her touch.  In the cell across the way, they find AS.  She takes him and gives him to Zedd.  They return to the queen and tell her that if she comes back to find the same situation, she will remove the queen from power.  She asks to see Giller.   The queen tries to make excuses, but eventually gives in.  They are taken upstairs to find Giller’s body hung.  He used his life fire to save someone (Rachel).  They give the queen their leave and take off, pretending to be happy that Giller is dead.  They all feel defeated because they don’t know what to do next.
   
   40.  Chase approaches them with four horses and Rachel.  He saved the little girl from a gar and was able to get her shoes and a cloak.  Richard notices the bread is still with her.  Rachel seems to know Siddin.  Chase takes them back to their camp from the previous night where he has a wild boar roasting.  On the way back, Chase explains that he went to Michael, Richard’s brother, and he commanded that his men come to help them.  Brophy comes around and knows Rachel.  The wolf tells her that Kahlan isn’t scary like she thinks. Kahlan explains why she thought Giller was bad.  When Richard asks, Rachel gives up her loaf of bread and he reveals the box.  Chase asks Rachel to join his family and she agrees.  As they discuss what to do next, Zedd points out that Richard has a spell over him that the queen’s artist had put on him.  He intends to give them the box and go back to have the spell taken off.  Kahlan kisses him before he leaves.   Richard finds the cave, but it is too dark to see all the drawings that represent the spells.  He uses the night stone to find the one of him.  He puts the night stone away and realizes the drawing is too big to erase the entire thing.  The artist, James, appears and brings a torch with him.  Richard starts altering the drawing.  James starts a new one to stop him, but dies in the process.  Richard leaves the cave and is surrounded by a woman and a horde of men.  He goes to fight them and the woman takes his power.  Denna, a Mord-Sith, causes him extreme pain and he is defeated.  Darken Rahl sent her after him.  She uses her agiel against him, but Richard still resists her.
   
   41.  Richard wakes up in an unfamiliar place with all of his belongings. Denna has been sent by Darken Rahl to train him through pain.  Rahl wants to ask Richard a question and Denna is going to make sure that he answers.  She says that one of his friends betrayed him to Rahl.  Richard realizes that Rahl wants the Book of Counted Shadows.  He will try to read Richard’s entrails to find it and will then realize that it is only in Richard’s head.  He locks away part of himself that includes his dignity.  He can only make the pain stop by thinking of something pleasant about Denna.  She chains Richard in the wall to start training him with the agiel, which causes pain with even the slightest touch.  She leaves him like that overnight, warning him not to sleep because if he falls off of the peg that holds him up, he will spend the entire night in pain.  The next morning, Denna cleans him up so they can go to dinner with the queen.  Princess Violet is allowed to use Denna’s agiel on Richard.  She starts telling him what she is going to do when Kahlan returns and sticks her tongue out at him.  Richard punches her in the jaw and her teeth sever her tongue.  The guards take Violet away.  Denna puts the agiel in her own mouth and kisses Richard to bond them.  She has decided to take him as a mate.  She explains that her training began when she was 12 and didn’t end until she was punished for 2 years for killing her master.  Richard asks her to help him stop Darken Rahl and says he will stay with her willingly if she does.  She commences her torture.
   
   42.  When Denna prepares to leave with Richard, the queen confronts her and demands the seeker’s life.  Denna uses her agiel on the queen, who has a heart attack and dies.  She allows Richard to sleep in the carriage on the way to D’Hara were the Mord-Sith are trained.  She introduces him to Constance, who helps Denna with his training.  One night, Denna explains that since her agiel was used to break her, it hurts her whenever she uses it.  Richard surprises her by asking Constance to finish his training.  She beats him bloody before asking why.  He explains that he doesn’t want to cause her any pain.  She starts being nicer to him.  When Constance asks to help her, she refuses and says that she is going to take him that night.  She allows him to sleep in the bed, saying that she needs the floor for a sore back.  The next morning, she takes him for a walk around the huge compound.  He enjoys the trees and can’t seem to remember why.  Denna explains that he will forget things from before her training.  The training after that isn’t nearly as rough.  One evening, Darken Rahl summons Denna and she leaves Richard with Constance.  When Constance is done with him (keeping to what Denna told her), she tells Richard to wait for Denna.  She returns much later and asks him to help her bathe.  She is covered in welts from Darken Rahl for being to easy on the seeker.  She tells him that Rahl will see him the next day.  Richard uses the supplies in his pack to make a healing cream for her.  She tells him that to become a Mord-Sith, a caring girl is chosen and then broken how she did to Richard.  Then her trainer breaks her mother and eventually kills her.  Lastly, the girl has to break her father and kill him.  Rahl personally trained her on the last breaking.  Richard is horrified that Denna had to go through that.  She asks to be with him without the agiel.
   
   43. The next morning, Constance makes some comments about Denna that Richard doesn’t like.  He threatens to kill her.  Denna takes him in front of Darken Rahl to answer his questions.  Richard feels a power in him that enables him to kill the guards.  When Rahl asks about the Book of Counted Shadows, Richard tells him that he burned it and the knowledge is only in his head.  He recites the first page, which mentions that a confessor must use her power to know if the book is real.  Richard is suddenly reminded of Kahlan and the part of him that was locked away breaks free.  Rahl asks him how he partitioned his mind, but Richard does not know.  Rahl sends Denna away, explaining that it is not her fault that he was able to partition his mind.  He tells Richard that George Cypher was not his father, but he doesn’t know who it really was.  There is a wizard’s web over him preventing him from telling.  Rahl explains that he has other ways of finding out how the boxes work and has already taken the cover off of one.  Richard proves that he does have the book memorized by taking the cover off of the second.  Rahl explains that Kahlan and the wizard are on their way with the box.  Richard doesn’t believe him.  Rahl lets him go, knowing that Richard will come back when it is time to open the first box because he’ll want to save Kahlan.  He warns Richard that once he leaves the room, Denna will have power over him again.  Richard must kill her to free it. Rahl casts a wizard’s web on him to make him look like his friend’s greatest enemy.  He goes to devotion to clear his mind and figures out how to use a white rage.  When he returns to Denna, she is waiting to be killed.  Rahl told her that if Richard is the seeker that can turn the sword white, he will be the one to kill her.  She requests that he take her agiel and wear it around his neck.  He kisses her and kills her with the sword.  
   
   44. Constance finds Richard and tries to get him to use his sword against her.  Richard cannot forgive her completely to be able to turn the sword white and kill her, so he uses Denna’s agiel on Constance.  He finds his way out of the castle and wonders where to go.  The guard offers him a horse and tells him how far away his friends are.  Richard doesn’t know how much he can do if he must return in a week.  The guard warns him about Rahl’s red dragon that is on the way.  Richard takes off thinking about the dragon.  After riding a long time, he finds her and she eats his horse.  The dragon, Scarlet, plans to eat him, but he distracts her.  He offers to get her egg back from Rahl if she will help him.  Richard thinks of a plan to get the egg away from the dozen or so gars that are protecting it.  Zedd tries to find the night stone and starts being pulled to the underworld.  He asks Kahlan to touch him to break its hold on him.  Zedd is unaffected by her touch, but it serves as a beacon to help him out of the underworld.  He senses that the night stone was there, destroyed by Rahl, but couldn’t tell if Richard was.  
   
   45. Richard and Scarlet land by a cave a little away from the gars guarding the egg.    Scarlet lights a torch for him and he goes into the cave.  He finds a small hole with air coming through.  He starts making his way through and gets stuck.  He has to calm himself by chanting his devotions that he learned at the D’Haran palace.  Something crawls up his leg and bites him.  He forces himself through the opening and is bloody from the bite and squeezing through the opening.  He rips off the blood soaked part of his pants and throws it to the blood-flies using a rock.  He starts maiming the gars to add to the confusion.  Once he has the egg, Scarlet comes and carries him in her claw.  She flies them away to a hidden place that Rahl will not be able to find.  Richard tells her what happened.  She admits that she came after him because she thought he was going to run away.  She heats up a rock to keep her egg warm and they are ready to start looking for his friends.
   
   46. They don’t have any luck finding Richard’s friends. Scarlet brings back a deer.  The next day, he sees them on a trail.  Zedd instantly uses his wizard’s fire on Richard, thinking that he is Darken Rahl from the web put on him. Richard tries to talk to them, but they only hear him speaking high D’Haran.  Eventually, he gives up and leaves to find his brother.  Scarlet drops him off ahead of the army and leaves Richard for the night.  Michael and fifty of his personal guards ride ahead.  Richard has to chase after them and sneaks around to find Michael and give him the salute he used to when they were children.  When he finds Michael, he greets him as his little brother.  Richard knows that he is the one that betrayed him.  He asks for the box and Michael says that Zedd has it.  Richard cannot kill his brother, so he leaves.  Michael tries to have his guards stop him, but Richard grabs a horse and takes off.
   
   47. Zedd wakes up the next morning and can’t help thinking about how strangely Darken Rahl had acted the previous day.  He is extremely worried that he saw Rahl with the sword of truth and even more so that he turned it white.  The prophecy says that only the one who turns the sword white can close the veil to the underworld.  Demmin Nass arrives with his men to take Kahlan.  Zedd is frozen in place.  Kahlan is able to touch one of them.  Two of them hold her down while another starts to rape her.  Chase rushes him and knocks him down, but Demmin knocks him out.  Brophy comes to attack Demmin, but the man pulls a knife out and kills the wolf.  Kahlan goads Demmin into doing whatever he can to her and what has happened to Richard.  Kahlan starts to scream and it combines her power of love with its counterpart from the underworld.  Darkness is everywhere until Kahlan gets up and is surrounded by blue light.  She frees Zedd, who then frees Chase.  Zedd explains the Con Dar, the blood rage, is something that only the strongest Confessors can do, but must be taught by the mother.  Kahlan should not be able to do it.  She is going to try to kill Darken Rahl even if it is futile.  She tells the man she touched to die and his does.  She uses her power on Demmin.  Kahlan cuts off his testicles and makes him eat them.  She asks if what he told her about Richard is true.  He says that it is and adds that Denna took him as her mate and probably wouldn’t have survived that long, but he did not see him die.  Kahlan ask for poisonous berries and something to crush them with.  Chase buries Brophy while Kahlan applies the berry paste to her face.  She pulls out her confessor’s dress and puts it on before taking off.  Chase wants to go with them to help Richard, even after Zedd explains what a Mord-Sith is.  Since Chase doesn’t have any magic, he doesn’t have anything to worry about.  Zedd has protection against them, and Kahlan’s touch is deadly to the Mord-Sith.  They catch up with Kahlan and follow her.  
   
   48.  Richard rides, but cannot find his friends.  He sleeps under a rock for the night.  When he wakes up, he realizes that he is farther from the People’s Palace than he thought.  He goes to get on his horse when heart-hounds start attacking.  Scarlet comes to rescue him and takes him to the People’s Palace.  Richard says goodbye to the dragon and then is welcomed back into the palace.  He goes to devotion to avoid trouble.  Rather than chanting about Darken Rahl, he changes the words to be about Kahlan.  He knows what to do and gets up to leave.  When two Mord-Sith try to stop them, he explains who he is and how he killed Denna.  They let him pass and he makes his way to Darken Rahl’s garden.  Zedd and Chase follow Kahlan into the People’s palace.  The Mord-Sith run from Kahlan as she approaches.  Zedd directs her where to go.  The entire palace is a spell constructed to protect Rahl with the Garden of Life in the center.  He can also feel where the boxes of Orden are since two are uncovered.  But he worries because he senses them three times as strong as before.
   
   49. Zedd and Chase follow Kahlan into the garden to find Darken Rahl already there.  He seems afraid of the Mother Confessor and pulls out his sword.  He tries to talk to her, but she can’t understand him.  Zedd realizes that he is actually Richard too late; Kahlan has already used her touch on him.  The real Darken Rahl appears, blocking the others from getting to him.  The Con Dar fades from Kahlan as she realizes what she has done.  Rahl promises that he will not hurt Kahlan if he recites the book.  Michael holds back Kahlan with a knife and won’t let her answer.  Richard speaks the book out loud.  Rahl does as the book instructs him to until the very end.  The last part tells him to open the box with two shadows.  Rahl says that the information he got from another source said that was the box that would kill him and threatens to kill Kahlan.  Richard screams that he is telling the truth.  Rahl believes him and opens the box.  The power of Orden sweeps over him.  Zedd feels completely defeated, until things start to change.  Rahl has opened the box that will kill him.  Richard explains that he wasn’t really touched by Kahlan’s power.  Rahl had already taught him how to use the other side of the magic.  Richard left out the last step of choosing which box to open.  He goes after Kahlan.  Zedd stays with Rahl, who asks if Richard is the wizard’s son.  Zedd explains that RIchard is is grandson.  When Rahl raped his daughter, Richard was born.  An explosion engulfs Rahl in the black pit as he swears it isn’t over.  Zedd removes the webs and the people know that Richard is the new leader of the Rahl house.  He will eventually have to tell him that Darken Rahl is his father.  Richard Finds Kahlan ready to kill herself.  He stops her and explains that her power didn’t work on her because he already loved her completely.  There was nothing for the power to distort.  Zedd knew the whole time, but wasn’t able to tell them because it could spark doubt.  He warns them not to tell anyone.  Richard asks if it is over with Darken Rahl.  Zedd says that he is dead.  Michael is furious with Richard; he thinks that Darken Rahl was going to end people’s suffering.  Richard pulls out the sword of truth to execute Michael, but cannot make it turn white because he can’t forgive what his brother has done to others.  He tells Chase to execute him.  When Richard sees his brother’s head on a pike, he orders it taken down.  The army’s and guard’s leaders swear loyalty to him.  Richard starts giving out orders to help people.  He wants all the agiels destroyed and the Mord-Sith set free and rehabilitated into society.  Scarlet comes along to tell Richard that her egg hatched.  She wants him to come see the baby dragon.  Richard asks for her to take Siddin back to his parents afterwards.  She agrees.  Zedd plans to return to Aydindril and tells Richard to come see him when he is ready.  The master of the Panis Rahl’s crypt comes and tells him there is trouble with the walls melting.  Zedd gives him instructions on how to seal the tomb shut to prevent the palace from melting.  He asks for lunch before he plans to go and take Adie back to Aydindril with him to read about the underworld.

Significance of the Study

The basis for the television show, Legend of the Seeker.

In the aftermath of the brutal murder of his father, a mysterious woman, Kahlan Amnell, appears in Richard Cypher’s forest sanctuary seeking help . . . and more. His world, his very beliefs, are shattered when ancient debts come due with thundering violence.

In a dark age it takes courage to live, and more than mere courage to challenge those who hold dominion, Richard and Kahlan must take up that challenge or become the next victims.

Beyond awaits a bewitching land where even the best of their hearts could betray them.

Yet, Richard fears nothing so much as what secrets his sword might reveal about his own soul.

Falling in love would destroy them–for reasons Richard can’t imagine and Kahlan dare not say.

In their darkest hour, hunted relentlessly, tormented by treachery and loss, Kahlan calls upon Richard to reach beyond his sword–to invoke within himself something more noble.

Neither knows that the rules of battle have just changed . . . or that their time has run out.

The post Write up on Terry Goodkind’s Wizard’s First Rule appeared first on DayDreamin’ Comics.

Betrand is born of rape and from a young age exhibits some strange physical characteristics like hairy palms, connected eyebrows and a lack of wit that holds him back from his peers. He is constantly ridiculed and picked on as he grows up and one day succumbs to a rage that he not only can’t remember clearly the next day, he believes what he can recall to have been a dream. It begins with the taste of blood, then graves are disturbed, and eventually the body count begins to mount as his uncle and caretaker realizes something is terribly wrong with the boy. A wolf hunt ensues with no real success, traditional weapons seem useless, and soon the wolf is shot by (Uncle) Aymar with a silver bullet. The body of the wolf is not recovered, and the next day Aymar digs his silver bullet out of the leg of young Bertrand. Bertrand learns he’s able to keep the wolf at bay by partaking of the blood of his willing young lover Sophie, but he soon loses himself in Paris amidst the turmoil of civil war and moves toward his fate.

One of the most interesting facets of the novel is the character of Aymar and his love/hate relationship with Betrand. Aymar muses multiple times by the end that yes, Bertrand is a monster. Yes, he kills those around him, but Betrand’s sins are nothing compared to the death toll humanity inflicts on itself daily, especially during the conflict Paris is currently undergoing. Is Betrand really that bad? The internal struggles by both Aymar and Bertrand serve to highlight how we abhor certain things but accept much, much worse as par for the course in life.

 It does not take Aymar long to realize that something is decidedly wrong with his young charge; in fact, Caillet is a werewolf, who loves nothing more than leaping out of his bedroom window at night and killing livestock and assorted wayfarers around the countryside. Years later, as a young man, Caillet runs away to Paris, to continue his depredations in a more populous arena, but at a most inauspicious time: right in the midst of the Franco-Prussian War, and right before the incredible violence of the Paris Commune of 1871.

But this novel is so much more than a simple tale of horror, although there are many grisly scenes. Endore (whose real name was Harry Relis) views his werewolf not as a monster, but rather as a sympathetic victim. Although Bertrand commits some truly horrible acts — killing his best friend, committing incest with his mother, despoiling graves, murdering countless creatures, draining his wealthy Jewish girlfriend (a neurotic, self-destructive, death-obsessed girl who today would probably be a Goth) slowly of her life’s blood — the author makes it clear that the atrocities going on around him (e.g., the 20,000 Parisians killed by the Versaillists during the Commune) make his sins seem small indeed. Presciently, the author says that future wars will kill millions, a prediction sadly borne out just a decade after this book’s release.

Significance of the Study

In The Werewolf of Paris by Guy Endore, we are transported to 19th century France, where we meet Bertrand Caillet, a young man who discovers the horrifying truth about his lineage. His mother, a prostitute, was raped by a werewolf, and he is the product of this monstrous union. Bertrand’s life is filled with struggle and suffering, not just from the outside world, but from his own inner turmoil as he grapples with his dual nature.

Endore paints a vivid picture of the political and social unrest in France during this time, intertwining Bertrand’s personal struggles with the larger issues of the day. The werewolf is a metaphor for the primal, untamed forces that lurk within us all, and Bertrand’s internal battle with his bestial nature mirrors the external turmoil of a nation on the brink of revolution.

Unleashing the Beast

As the story progresses, Bertrand’s inner werewolf is unleashed with increasing frequency, leading to a string of brutal murders. His actions draw the attention of the authorities and the public, and he becomes a notorious figure, feared and reviled by all. Despite his best efforts, Bertrand cannot control his animalistic urges, and the line between man and beast begins to blur.

Endore uses Bertrand’s transformation into a werewolf as a metaphor for the dehumanizing effects of war, political upheaval, and social injustice. Bertrand, a victim of his circumstances, is pushed further and further into his bestial nature by the brutal world around him, mirroring the descent of society into chaos.

Love and Loss

Amidst this backdrop of violence and bloodshed, Bertrand falls in love with Sophie, a young woman who is drawn to him despite his monstrous reputation. Their love offers Bertrand a glimmer of hope, a chance at redemption. However, his werewolf instincts eventually drive him to kill her, shattering his last hope of salvation and pushing him even further into despair.

Endore’s portrayal of Bertrand’s doomed love affair is a poignant exploration of the destructive power of the werewolf’s curse. Despite his best intentions, Bertrand cannot escape his true nature, and his attempts at normalcy only lead to more pain and suffering.

The Final Transformation

As the story nears its end, Bertrand is captured and sentenced to death. In a final act of defiance, he embraces his werewolf nature completely, refusing to beg for mercy or show any remorse for his crimes. On the night of his execution, Bertrand transforms into a werewolf and escapes, leaving a trail of death and destruction in his wake.

In the end, The Werewolf of Paris is a tragic tale of a man torn apart by the conflicting forces within him, a victim of his own nature and the cruel world around him. Endore’s powerful narrative and vivid imagery leave a lasting impression, forcing us to confront the darker aspects of our own humanity.

The post Write up on Guy Endore’s werewolf of paris appeared first on DayDreamin’ Comics.

Black Lightning Year One No. 1 was written by Jen Van Meter, with art by Cully Hammer. It is the first of a 2009 six-issue limited series documenting the origins of the titular DC superhero.

Created by writer Tony Isabella artist Trevor Von Eeden, Black Lightning was introduced to comic readers in 1977. The first African American to headline a DC title, the hero, a man named Jefferson Pierce, has the ability to create and manipulate electricity. The character has been featured in animated film and series and video games, and headlined his own television program on The CW from 2018-2021.

 Hill is making the most of his five-issue tenure as writer of Detective Comics. He has introduced a brand new villain, teamed Batman up with Black Lightning, and laid the foundation for a new team of Outsiders. That’s a lot of bat for the buck, and that alone would make these stories worth checking out. But Hill has a secret weapon: he knows what makes Bruce Wayne tick. Despite all of the heavy lifting being done around the Bat-family in Detective Comics, Hill has a knack for putting a new spin on Batman’s well-worn drive and motivation, and he clearly has a tremendous love for the history of the character.

The first Outsiders came together by chance, shortly after Batman resigned from the Justice League. This was due to Wayne Enterprises CEO Lucius Fox being taken hostage after war broke out in the nation of Markovia. With the League being restricted from getting involved in international incidents, Batman couldn’t count on them in rescuing his friend.

Batman partnered with Black Lightning, but the two encountered unexpected allies in their battle with the despot Baron Bedlam. These included the assassin Katana, an amnesiac girl with light powers Batman named Halo, the element man Metamorpho, and the earth-shifting Geo-Force. Once Lucius Fox was rescued, Batman decided to continuing working with the new heroes, offering them his training and support.

https://www.comics.org/issue/30904

The post Write up on DC Comics Black Lighting “Historical Records” “First African American SuperHero appeared first on DayDreamin’ Comics.

1.1 Logical Programming

A logical operator is said to be truth-functional if the truth-values (the truth or falsity, etc.) of the statements it is used to construct always depend entirely on the truth or falsity of the statements from which they are constructed. The English words “and”, “or” and “not” are (at least arguably) truth-functional, because a compound statement joined together with the word “and” is true if both the statements so joined are true, and false if either or both are false, a compound statement joined together with the word “or” is true if at least one of the joined statements is true, and false if both joined statements are false, and the negation of a statement is true if and only if the statement negated is false.

Some logical operators are not truth-functional. One example of an operator in English that is not truth-functional is the word “necessarily”. Whether a statement formed using this operator is true or false does not depend entirely on the truth or falsity of the statement to which the operator is applied. For example, both of the following statements are true:

• 2 + 2 = 4.
• Conjunction: The conjunction of two statements α and β, written in PL as ┌(α∧β)┐, is true if both α and β are true, and is false if either α is false or β is false or both are false. In effect, the meaning of the operator ‘∧‘ can be displayed according to the following chart, which shows the truth-value of the conjunction depending on the four possibilities of the truth-values of the parts:

The basic rules and principles of classical truth-functional propositional logic are, among contemporary logicians, almost entirely agreed upon, and capable of being stated in a definitive way. This is most easily done if we utilize a simplified logical language that deals only with simple statements considered as indivisible units as well as complex statements joined together by means of truth-functional connectives. We first consider a language called PL for “Propositional Logic

The validity of this argument can be made more obvious by representing the chain of reasoning leading from the premises to the conclusion:

1. Either cat fur was found at the scene of the crime, or dog fur was found at the scene of the crime. (Premise)
2. If dog fur was found at the scene of the crime, then officer Thompson had an allergy attack. (Premise)
3. If cat fur was found at the scene of the crime, then Macavity is responsible for the crime. (Premise)
4. Officer Thompson did not have an allergy attack. (Premise)
5. Dog fur was not found at the scene of the crime. (Follows from 2 and 4.)
6. Cat fur was found at the scene of the crime. (Follows from 1 and 5.)
7. Macavity is responsible for the crime. (Conclusion. Follows from 3 and 6.)

1.2 Datalog

Datalog, a logic programming language defined in the early 1980s, was originally used to describe and implement deductive databases [GMN84]. Given a set B representing the data in the database, a Datalog program P specified recursive views that a user could then query. Logically, B is a set of ground atomic formulas (facts) and P a set of implicational formulas of a restricted form (Horn clauses).

The semantics of such a Datalog system, as well as some actual implementations [CGT89], was given by the set of the atomic logical consequences of P with respect to B, in symbols {p(~t) | P, B ` p(~t)} — a beautifully pure use of logic. In the last ten years, Datalog has had a resurgence in a variety of domains other than databases: to provide user authentication [LM03], to implement network protocols [GW10, LCG+06], to program cyberphysical systems [ARLG+09] and to support deductive spreadsheets [Cer07], just to cite a few. In these applications, the contents of the fact base B is highly dynamic, at least compared to the relatively static databases of the 1980s.

 This requires efficient algorithms to assert new facts and to retract stale facts. While such algorithms have been proposed [ARLG+09, CARG+12, GMS93, LCG+06, NJLS11], the connection with logic has lagged behind, especially for retraction. Indeed, although it is easy to give a logical specification of assertion and retraction, as done in Figure 1, a direct implementation of these specifications is horribly inefficient as it involves recomputing logical consequences from scratch after each assertion or retraction. In this report, we propose a logical interpretation of Datalog that supports updates internally.

Like with the traditional reading, the semantics of a Datalog system (P, B) will be the set {p(~t) | P, B ` p(~t)} of its logical consequences (plus some bookkeeping information). However, the assertion or retraction of a fact a will trigger a small number of logical inferences to determine the new set of logical consequences rather than performing a full recalculation. In this way, the proposed encoding not only constitutes a logical specification of the static and dynamic aspects of a Datalog system, but it can also be viewed as an efficient algorithm to compute updates. Because retraction results in the removal of inferred facts, we base our encoding on a fragment of first-order linear logic rather than on the traditional predicate calculus. The main contributions of this report are therefore

1) the definition of an encoding of Datalog which models the assertion and retraction of facts within logic, and

 2) a proof of its correctness with respect to the traditional interpretation.

Proof Theory Another approach to find whether a logically follows from P is to apply a sequence of logically sound inference steps starting from P and ending with a.

The sequence of steps from P to a is called a proof, and serves to prove that a can be logically derived from

P. P ⊢ a is the fancy way to write this.

This states that there exists a proof of a from P. There are a number of questions to address regarding our proof system. First, what are the types of inference steps that are permitted? Once we have established that, we must address soundness and completeness.

• Soundness:

 For any P and a, if P ⊢ a, then P |= a.

• Completeness: For any P and a, if P |= a, then P ⊢ a

First-order propositional logic is sound and complete. That is, anything provable in it is in fact true, and anything true with respect to it is provable. It is one of the greatest mathematical results of the 20th century that first-order logic (predicate calculus) without arithmetic is sound and complete. (G¨odel’s Completeness Theorem) It is arguably the greatest mathematical result of the 20th century that first-order logic (predicate calculus) with full arithmetic and second-order logic are necessarily incomplete. (G¨odel’s Incompleteness Theorem)

1.3 Logic as a Database? Problems?

Horn Programs What are the problems with using CNF propositional logic for our knowledge-bases / databases / programs?

1. Looks really unnatural.

 2. Allows for meaningless databases / programs (that is, that have no models).

3. Where’s the data?! To address both points 1 and 2, we shall restrict the types of clauses permitted

Logic as a Database! Datalog We call a program that consists of just rules and facts—Horn clauses each with exactly one positive proposition—a Datalog database. We write queries as Horn clauses that contain no positive proposition. A query can be evaluated against a Datalog database as a resolution refutation proof.

Horn clause: At most one positive proposition appears. Horn clauses are more natural, and have a correspondence to database concepts.

Rule / View: a ← b, c. (a ∨ ¬b ∨ ¬c)

Fact: b. (b)

Query: ← a. (¬a)

• Easy to read when rewrittin as implications.

 • Every database / program is meaningful; that is, consistent! (Really?! . . .)

Logic as a Database! Datalog

We call a program that consists of just rules and facts—Horn clauses each with exactly one positive proposition—a Datalog database.

 We write queries as Horn clauses that contain no positive proposition. A query can be evaluated against a Datalog database as a resolution refutation proof.

query evalutation ≡ proof

So how do we know an answer to a query (with respect to the database) is correct?

Its evaluation is equivalent to a proof that it is correct (that it is a logical consequence)! We still need to address point 3, “Where’s the data?!”

We shall need to use (first-order) predicate calculus logic instead of just (first-order) propositional logic.

How do we know? Consider the interpretation in which we assign true to every proposition. Next, consider each clause: There is exactly one positive proposition per clause in a Datalog database, by definition.

 Thus every clause is true with respect to the all-true interpretation, and thus the all-true interpretation is a model. Of course the all-true model is not so interesting. . .but it does guarantee that any datalog database is consistent

1.3a Reasoning about Queries & Databases

Datalog and its logical foundations—model and proof theories—provide us with tools to address other general questions about queries and databases.

 • Given two Datalog queries, are they equivalent with respect to the database? That is, must they evaluate to the same answers?

• Does there exist a query of a given question we have in mind? That is, is the question even askable in Datalog? With this database?

• Given to Datalog databases, do they represent the same data? Is any question possible to state for one of them also possible to state for the other one?

The Move from Propositional Logic to Predicate Calculus

• Add arguments to propositions. Now call them predicates.

• Add logical variables. (For use in rules and in queries.)

• Add quantifiers for the variables: ∀ and ∃.

E.g., grandmother (GM, X) ← moth

Datalog permits only universal-quantified clauses. Thus no explicit existential-quantification is allowed.er (GM, P), parent (P, X).

Datalog Database versus “Prolog” Program We generally call a Horn-clause predicate calculus “theory” that we have written down a logic program. The Prolog programming language’s syntax looks just like this. So when do we call it a Datalog database instead? If it uses logical function symbols, it is considered a program. If it does not, it is considered a database. This is the logical distinction between them.

Logical function symbols?? This is essentially a data-structure, such as a list or record, that we could use as an argument to a predicate instead of just a simple value.

 • grandmothers ([lallage, ruby, sally ] , parke)

• product (#13, widget (a, b) , $23.50) Function symbols are needed for arithmetic. (We usually add a limited form of arithmetic to a fuller Datalog.)

Here’s an example of a simple Datalog program that defines the ancestor relation recursively:

Figure 1. Grandparent Table

A clause is a head literal followed by an optional body. A body is a comma separated list of literals. A clause without a body is called a fact, and a rule when it has one. The punctuation :- separates the head of a rule from its body. A clause is safe if every variable in its head occurs in some literal in its body. The following are safe clauses:

Figure 2. Parent Table

1.4 How Does Datalog Work

At its core, Datalog is a set of rules that define relationships between objects in the form of predicates. These predicates can then be used to perform queries and manipulate data. Predicates are represented as facts, which are simply statements that define the relationship between entities.

For example, let’s say we have a database of people and their relationships to one another. We can define predicates for names, genders, parent-child relationships, and so on. These predicates can then be used to perform queries such as “who are the parents of Joe?” or “who are Joe’s grandparents?”.

Datalog rules define new relationships between existing entities. For example, we can define a rule that says “if X is a parent of Y and Y is a parent of Z, then X is a grandparent of Z”. This rule can then be used to derive new insights from the existing database, such as finding all of the grandparents of Joe.

1.5 Modern Applications of Datalog in Database Management Systems

Datalog is becoming increasingly popular in modern database management systems, especially those that deal with large volumes of data. Its versatility, performance, and scalability make it the perfect tool for managing complex databases and performing complex queries.

One of the most powerful applications of Datalog is in graph database management systems. Graph databases are designed to handle complex, interconnected data and relationships, which make them ideal for applications such as social networking, recommendation engines, and fraud detection.

Datalog is particularly well-suited for working with graph databases because it allows you to express complex relationships between entities in a simple and intuitive way. You can define rules that traverse the graph and derive new relationships between entities, allowing you to perform complex queries with ease.

Another popular application of Datalog is in data warehousing. Data warehouses are designed to store large quantities of data and support complex queries that analyze that data. Datalog is perfect for data warehousing because it allows you to define relationships between entities in a way that’s easy to understand and optimize. You can define rules that summarize and aggregate data across multiple dimensions, allowing you to gain unique insights into your data.

Finally, Datalog is also being used in modern machine learning systems. Machine learning algorithms rely heavily on data management and processing, and Datalog’s ability to handle large volumes of data, perform complex queries, and define recursive relationships make it an ideal tool for this application.

The post Write up on Tech Geek History : DataLog & Logical Programming appeared first on DayDreamin’ Comics.

Derrick Albert Bell, Jr., was born in Pittsburgh, Pennsylvania, on November 6, 1930. Bell was offered a scholarship to Lincoln University but was unable to attend because he did not receive enough financial aid. Becoming the first member of his family to go to college, Bell chose to attend Duquesne University, earning his A.B. in 1952.

While attending Duquesne University, Bell joined the ROTC, and following his graduation, went to Korea as part of the U.S. Air Force. Returning from the war in 1954, Bell attended the University of Pittsburgh Law School, earning an L.L.B. in 1957. Bell was hired by the U.S. Justice Department after graduation, but left in 1959 over his refusal to terminate his involvement with the NAACP; subsequently, Thurgood Marshall recruited him to join the NAACP Legal Defense Fund where he oversaw three hundred school desegregation cases. In 1966, Bell was named deputy director of civil rights at the U.S. Department of Health, Education, and Welfare, before becoming a teacher at USC law school and director of USC’s Western Center on Law and Poverty in 1968.

In 1971, Bell became the first African American to become a tenured professor at Harvard Law School; there he established a course in civil rights law and wrote Race, Racism and American Law, which today is a standard textbook in law schools around the country. Leaving Harvard, Bell became the first African American dean of the University of Oregon Law School, and in 1985, he resigned in protest after the university directed him not to hire an Asian American candidate for a faculty position. Returning to Harvard Law School, Bell would again resign in protest in 1992 over the school’s failure to hire and offer tenure to minority women.

In addition to his work in the classroom, Bell was an acclaimed author, having written numerous books, most notably his series featuring fictional civil rights leader Geneva Crenshaw, including And We Are Not Saved and Faces at the Bottom of the Well. In 2002, Bell wrote Ethical Ambition: Living a Life of Meaning and Worth, which contained his thoughts on achieving success while maintaining integrity. Most recently, Bell authored Silent Covenants: Brown v. Board of Education and the Unfulfilled Hopes for Racial Reform. Bell had been the recipient of numerous honors and awards; his later work included serving as a visiting professor of law at the New York University School of Law.

Derrick Bell passed away on October 5, 2011 at age 80.

Literature Review

Dystopian satire. Aliens visiting the US offer to provide the country with everything needed to solve many of its problems in exchange for the entire African American population. Offered enough gold to bail out the country, chemicals that can un-pollute the environment, and a safe nuclear energy and fuel to overcome the energy crisis, the US government agrees, and all African Americans are loaded onto what are obviously slave ships.

For the first time in history, extraterrestrial beings have announced an impending visit to the United States. The whole population watches as ships land on the East Coast and the aliens walk on water toward them. They make a simple announcement: they have gold to solve the country’s debt crisis, de-pollutants to solve the climate crisis, and safe nuclear energy, all of which they are prepared to offer to the country – in exchange for the country’s Black citizens.

Immediately legislatures are called into session to discuss the proposal. Millions of callers give their opinions – most whites in favor of the trade, most blacks not. The President’s cabinet members offer various justifications for the trade – the overwhelming benefit of the aliens’ offerings, the lessened burden on social services without Black people, the alleviation of racial tensions, and the argument that leaving would be a patriotic duty any of them would take on if asked. The President calls on Gleason Golightly, a conservative Black economics professor, to weigh in. Golightly appeals to the moral side of the argument, backing his vehement opposition to the mass exile with a review of the many times in American history leaders considered and rejected the proposition of sending Black people away. The Cabinet did not budge in their opinions, and Golightly was left dejected at his mistake, having always considered himself one who could win over white politicians with cunning.

An Anti-Trade coalition drafts bills and responses to counteract the proposal, as well as a plan to evacuate and hide Black people in the worst-case scenario. Golightly speaks at their convention with the argument that they should accept the trade with enthusiasm and convince white people that Black people would be receiving a better future – a prospect which would enrage white people and reverse the trade. Again, his proposal is rejected.

Fortune 500 CEOs balk at the prospect of losing the profit they make off Black people, and launch massive media campaigns with images of mixed-race couples, appealing to the country’s pathos. Jewish organizations condemn the trade as analogous to Hitler’s ‘final solution’, but are intimidated out of their organizing plans by the government. The Televangelists of America herald the call from the aliens as God’s will, and organize a rally depicting dutiful Black people boarding a ship resembling a sacrificial alter. Groups in favor convene a constitutional convention and introduce a twenty-seventh amendment, requiring ‘special service’ calls by Congress to be headed by all citizens, to be ratified. They proclaim that the Framers’ intention was to build an all-white country, and attempts to the contrary have failed. While pro- and anti- trade groups riffed back and forth in the media, the Supreme Court and public opinion sided with pro-trade, and the amendment was ratified. Thousands of Black people were killed trying to escape. On Martin Luther King Day of that year, the Black population was rounded up, stripped, and sent into the spaceships.

Critical Race Theory has come to dominate how many modern people think about race and racism. I’ve mentioned this theory before, but have only offered a partial definition. Personal time doesn’t allow me a full explanation of the origins of Critical Race Theory (CRT), even though I think CRT is an important topic worthy of discussion. But it so happens that one of the founders of CRT, Derrick Bell, crafted a science fiction story called The Space Traders intended to illustrate the plight of black people in modern America. This post is a discussion of that science fiction short story and how its backstory stemmed from a number of the ideas central to CRT. And…by the way…we are finally on a subject that’s clearly on topic for Speculative Faith–science fiction!

Significance of the Study

The central character in The Space Traders is Gleason Golightly, a conservative black man who is an economic advisor to a white Republican president. As I’ve said, he’s the only character in the story with any depth. Though the story doesn’t begin with him.

The tale starts in a near-future scenario–from the point of view of the Nineties–in which aliens mysteriously arrive on January 1st and offer the United States unlimited clean energy and vast wealth–with a catch. All the aliens want is the USA to hand over every single person in the country who identifies as black. For purposes unknown.

The tale begins on 1 January and counts up until 17 January–which in the story world is a Martin Luther King Jr holiday…and also the day the United States ushers all black citizens into the alien ship cargo holds. Irony blatantly intended.

The story says the aliens adopt a voice like the former president Ronald Reagan, which causes all white people to trust them but causes suspision in all black people in America. I’ll comment on this further down.

Action leaps to the White House and of course the conservative cabinet is all for making this exchange. After all, in this story world, their conservative policies have wrecked the environment and bankrupt the country–they despirately need the alien technology and wealth to rescue the country from its dire straits. (I won’t comment on this further, but yeah this bit of story backdrop is more than a bit over-the-top-politically-liberal.)

But they want to hear from their token black economics advisor–the story directly calls him “token” (and mentions black people see this character as an “Uncle Tom”). The concerns of the cabinet being: “How can we sell this exchange to the public?” and “How can we justify our actions?” and “How can make this work legally?” rather than, you know, the ethics of the situation.

The story says directly that if the aliens had asked for any other sub-group of people, such as green-eyed red heads, the exchange would never have been considered for a moment. I’ll comment on this in a bit.

Golightly naturally doesn’t want the exchange to take place. The story from this point forth is about his efforts to first avert the exchange but when his efforts fall through, to save himself by escaping to Canada.

His strategy to derail the exchange is that he meets with a group of black Civil Rights leaders and attempts to persuade them they should speak out in favor of the trade. If they could only show that black people are in favor of the prospect of being hauled away by mysterious aliens, then white people would themselves wonder why black people were for it, which would be their best chance to save themselves. Because, you know, if black people want the exchange, white people won’t want to give it to them. Reverse psychology.

Golightly fails to persuade the black community leaders. They think his idea is crazy–but he despairs when they don’t listen to him.

To legally get rid of all blacks, the USA needs to enact a constitutional amendment in which the military draft is applied to black people. So all will be drafted into “service” for the good of the rest of the country. Bell mentions a few legal parallels to this, specifically stating the US Constitution itself sacrificed black people for the purpose of making the USA, so the idea the nation would do so again isn’t unthinkable.

The story spends a bit of time talking about the Liberal opposition to the amendment allowing the exchange. Part of New York City is shown to be shut down for a while due to protests led by Rabbi Abraham Specter, with the cooperation of many American Jews. The story says at one point what the Jewish motivaton is–they don’t want to be the ones stuck at the bottom of the social order to receive the full ire of angry poor whites. So they want black people to stay.

Other, non-Jewish Liberals are referred to in that thirty percent of the USA is said to be against the amendment, while seventy percent are for it. Note that the US population of African Americans is about eleven percent, so nineteen percent of the rest of America were against the exchange. The Jewish population of America is less than nineteen percent and I can’t imagine Bell missing this detail, so clearly other, non-Jewish Liberals also opposed “the Trade.” The story never says who were the other Liberals, but it does assign them a motivation: Guilt.

.

Golightly attempts to make the case early on with the cabinet that white people will be extremely afflicted with a sense of guilt after the fact if they trade off all black people for the alien tech. This doesn’t affect the cabinet, but as far as the story is concerned, guilt is the only motivator to want to help black people other than not wanting to be at the bottom of the social order yourself.

The aliens warn the United States government that they if they let black people escape to other countries, the deal will be off. The story references a few getting away, but the USA mostly blocking any escapes. The federal government promises to send a few prominent black people to other countries in exchange for their help, including Golightly and his wife, but they take back this offer in the end.

The second to last moment of the story features Golightly trying to escape to Canada but being caught by the US Secretary of the Interior, a colleague Golightly worked with and knew personally. His wife wondered if black people would have been worse off if Golightly had managed to block the legislation, because black people would be blamed for the country not receiving aliens’ benefits. The story basically leaves this comment hanging.

In the end, the aliens require all black people to strip down to a single article of clothing before boarding the ship. The story pointedly says they leave the United States in the same way their ancestors had arrived. As ship’s cargo.

Other, non-Jewish Liberals are referred to in that thirty percent of the USA is said to be against the amendment, while seventy percent are for it. Note that the US population of African Americans is about eleven percent, so nineteen percent of the rest of America were against the exchange. The Jewish population of America is less than nineteen percent and I can’t imagine Bell missing this detail, so clearly other, non-Jewish Liberals also opposed “the Trade.” The story never says who were the other Liberals, but it does assign them a motivation: Guilt.

The post Write up on Derrick Albert Bell, Jr’s Space traders appeared first on DayDreamin’ Comics.